Oct 30 05:28:57.765211 ip-10-0-136-35 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 05:28:57.765224 ip-10-0-136-35 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 05:28:57.765230 ip-10-0-136-35 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 05:28:57.765513 ip-10-0-136-35 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 05:29:07.953265 ip-10-0-136-35 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 05:29:07.953286 ip-10-0-136-35 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 41f12e5ecf394d488122cb1e2a2a96ef -- Oct 30 05:31:09.776308 ip-10-0-136-35 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:31:10.217775 ip-10-0-136-35 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:10.217775 ip-10-0-136-35 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:31:10.217775 ip-10-0-136-35 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:10.217775 ip-10-0-136-35 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:10.222457 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222371 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.222457 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222453 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.222457 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222458 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.222457 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222461 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.222457 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222464 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222467 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222470 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222475 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222480 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222483 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222487 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222489 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222492 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222495 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222497 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222500 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222503 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222505 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222508 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222510 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222513 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222516 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222518 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.222667 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222521 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222523 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222526 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222528 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222531 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222534 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222536 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222539 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222549 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222552 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222554 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222557 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222560 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222562 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222565 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222569 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222571 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222574 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222577 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222580 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.223190 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222582 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222585 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222587 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222589 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222592 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222595 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222597 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222599 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222602 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222604 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222607 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222609 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222613 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222615 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222618 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222620 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222622 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222625 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222627 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.223696 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222630 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222632 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222635 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222637 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222641 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222645 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222648 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222651 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222655 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222658 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222662 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222664 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222667 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222670 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222673 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222675 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222678 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222680 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222683 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222685 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.224204 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222688 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222691 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222694 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222696 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222699 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222701 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222703 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222706 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222708 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222711 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222714 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222716 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222720 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222722 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222725 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222727 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222730 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222732 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222735 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222737 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.224749 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222740 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222743 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222745 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.222748 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223522 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223529 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223532 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223535 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223538 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223541 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223543 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223546 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223548 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223551 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223554 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223557 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223559 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223562 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223564 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.225267 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223567 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223569 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223572 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223575 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223577 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223580 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223582 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223585 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223587 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223590 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223592 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223595 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223597 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223600 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223603 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223605 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223608 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223610 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223613 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223615 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.225762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223618 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223620 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223623 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223625 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223627 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223630 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223632 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223634 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223641 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223646 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223649 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223652 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223655 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223657 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223660 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223663 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223665 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223668 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223670 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.226284 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223673 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223675 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223678 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223680 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223683 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223685 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223689 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223692 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223694 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223697 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223699 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223702 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223704 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223707 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223709 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223712 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223714 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223717 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223719 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.226767 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223721 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223724 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223726 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223729 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223733 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223735 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223738 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223740 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223744 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223746 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223749 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223751 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223754 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223756 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223759 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223761 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223764 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223766 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223769 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223772 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.227262 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223775 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223777 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223780 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223782 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223785 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223787 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223790 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223792 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223796 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223800 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223803 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223806 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.223811 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224333 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224339 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224343 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224347 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224350 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224354 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.227772 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224358 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224361 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224364 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224366 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224369 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224372 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224374 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224377 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224379 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224382 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224384 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224387 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224390 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224392 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224395 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224398 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224400 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224403 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224405 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224407 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.228281 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224410 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224412 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224415 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224418 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224420 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224424 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224428 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224431 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224435 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224438 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224441 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224443 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224446 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224448 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224451 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224454 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224456 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224459 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224461 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.228784 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224464 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224466 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224469 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224471 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224474 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224478 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224481 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224484 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224486 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224489 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224492 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224494 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224496 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224499 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224501 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224504 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224506 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224508 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224514 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.229277 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224516 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224519 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224521 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224524 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224527 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224529 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224532 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224534 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224537 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224539 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224542 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224544 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224547 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224550 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224552 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224555 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224557 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224560 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224564 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.229809 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224566 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224569 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224571 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224573 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224576 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224578 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224580 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224583 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224585 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224588 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224590 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224593 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224596 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224599 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224601 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224604 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224606 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224609 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224613 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224616 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.230415 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224618 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224620 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.224623 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226591 2578 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226603 2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226620 2578 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226630 2578 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226634 2578 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226638 2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226642 2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226646 2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226650 2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226653 2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226656 2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226659 2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226662 2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226665 2578 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226668 2578 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226671 2578 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226674 2578 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226677 2578 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226680 2578 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226683 2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226686 2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:31:10.230921 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226690 2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226693 2578 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226697 2578 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226699 2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226703 2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226706 2578 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226709 2578 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226712 2578 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226715 2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226718 2578 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226721 2578 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226725 2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226727 2578 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226730 2578 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226733 2578 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226736 2578 flags.go:64] FLAG: --enable-server="true" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226738 2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226743 2578 flags.go:64] FLAG: --event-burst="100" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226746 2578 flags.go:64] FLAG: --event-qps="50" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226749 2578 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226753 2578 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226755 2578 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226760 2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226762 2578 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226765 2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:31:10.231521 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226769 2578 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226772 2578 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226775 2578 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226778 2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226781 2578 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226784 2578 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226786 2578 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226790 2578 flags.go:64] FLAG: --feature-gates="" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226794 2578 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226797 2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226800 2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226803 2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226805 2578 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226809 2578 flags.go:64] FLAG: --help="false" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226812 2578 flags.go:64] FLAG: --hostname-override="ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226815 2578 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226818 2578 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226821 2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226824 2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226828 2578 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226831 2578 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226833 2578 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226836 2578 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:31:10.232182 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226839 2578 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226842 2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226845 2578 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226848 2578 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226851 2578 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226853 2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226856 2578 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226859 2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226861 2578 flags.go:64] FLAG: --lock-file="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226864 2578 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226867 2578 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226870 2578 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226875 2578 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226878 2578 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226881 2578 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226884 2578 flags.go:64] FLAG: --logging-format="text" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226887 2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226891 2578 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226893 2578 flags.go:64] FLAG: --manifest-url="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226896 2578 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226900 2578 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226903 2578 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226907 2578 flags.go:64] FLAG: --max-pods="110" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226910 2578 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226913 2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:31:10.232789 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226916 2578 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226919 2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226922 2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226925 2578 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226928 2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226935 2578 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226938 2578 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226941 2578 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226944 2578 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226946 2578 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226949 2578 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226952 2578 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226955 2578 flags.go:64] FLAG: --port="10250" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226957 2578 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226960 2578 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8f6d2c986d55497" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226963 2578 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226966 2578 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226969 2578 flags.go:64] FLAG: --register-node="true" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226971 2578 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226975 2578 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226978 2578 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226980 2578 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226983 2578 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226987 2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226990 2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226993 2578 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:31:10.233467 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226996 2578 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.226999 2578 flags.go:64] FLAG: --runonce="false" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227001 2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227004 2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227007 2578 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227010 2578 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227013 2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227016 2578 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227019 2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227024 2578 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227027 2578 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227030 2578 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227033 2578 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227036 2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227038 2578 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227052 2578 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227054 2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227060 2578 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227062 2578 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227067 2578 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227071 2578 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227075 2578 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227077 2578 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227080 2578 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227083 2578 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:31:10.234162 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227086 2578 flags.go:64] FLAG: --v="2" Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227090 2578 flags.go:64] FLAG: --version="false" Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227094 2578 flags.go:64] FLAG: --vmodule="" Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227098 2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227101 2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227234 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227238 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227241 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227244 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227247 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227249 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227252 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227254 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227257 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227260 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227263 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227267 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227270 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227273 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227276 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227279 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.234791 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227283 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227286 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227289 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227292 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227294 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227298 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227301 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227304 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227306 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227309 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227311 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227314 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227316 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227319 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227321 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227324 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227326 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227329 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227332 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227334 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.235365 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227337 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227339 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227342 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227344 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227347 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227350 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227352 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227356 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227359 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227361 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227364 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227366 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227369 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227372 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227374 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227377 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227379 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227383 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227385 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.235876 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227388 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227390 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227393 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227395 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227398 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227400 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227402 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227405 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227408 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227410 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227413 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227415 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227417 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227420 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227422 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227425 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227427 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227430 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227432 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.236376 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227435 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227438 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227441 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227444 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227448 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227451 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227454 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227456 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227459 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227461 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227464 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227469 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227471 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227474 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227477 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227479 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227482 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227484 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227487 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.236846 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227489 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227492 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227494 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227497 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227499 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227502 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227504 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227507 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227510 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227512 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227515 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227517 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227520 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.227547 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227584 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227589 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.237332 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227593 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227595 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227598 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227601 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227603 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227606 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227608 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227611 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227613 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227617 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227620 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227622 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227625 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227627 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227630 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227632 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227635 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227638 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227640 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227643 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.237744 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227645 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227648 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227650 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227652 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227655 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227657 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227660 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227662 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227664 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227667 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227671 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227673 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227676 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227680 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227684 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227687 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227690 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227693 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227696 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.238275 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227699 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227702 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227706 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227708 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227711 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227713 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227716 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227718 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227721 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227723 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227725 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227728 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227730 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227733 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227735 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227738 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227741 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227744 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227746 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227749 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.238750 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227751 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227754 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227756 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227760 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227763 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227765 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227768 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227770 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227773 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227776 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227778 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227781 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227783 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227786 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227789 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227792 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227794 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227799 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227802 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227805 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.239416 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227807 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227810 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227812 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227815 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227817 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227835 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227839 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227842 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227844 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227847 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227849 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227852 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227854 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227857 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227859 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227863 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227865 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227867 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227870 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.239918 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227872 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227875 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227878 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227880 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227884 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.227886 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.235986 2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.235999 2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.236014 2578 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.236018 2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236093 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236100 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236103 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236106 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236109 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236111 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236114 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236117 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236120 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.240417 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236123 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236125 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236128 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236130 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236133 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236135 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236138 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236140 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236143 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236146 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236148 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236151 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236153 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236156 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236159 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236162 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236165 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236168 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236170 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236173 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.240882 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236176 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236178 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236181 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236183 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236186 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236188 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236191 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236193 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236196 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236198 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236201 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236203 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236206 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236209 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236211 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236214 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236216 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236219 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236221 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.241403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236223 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236226 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236228 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236231 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236234 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236236 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236239 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236241 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236244 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236247 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236249 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236251 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236254 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236257 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236259 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236262 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236264 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236267 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236269 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236272 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.241886 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236274 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236278 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236281 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236285 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236289 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236293 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236297 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236299 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236302 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236305 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236307 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236310 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236312 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236314 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236317 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236320 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236323 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236325 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236328 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.242419 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236330 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236333 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236336 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236338 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236341 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236343 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236346 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236349 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236351 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236354 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236356 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236359 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236361 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236364 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236366 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236368 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236371 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236373 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.242877 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236376 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.236402 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236572 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236578 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236581 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236583 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236586 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236589 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236591 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236594 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236596 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236599 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236601 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236604 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236606 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236609 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.243403 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236611 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236620 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236623 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236625 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236629 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236631 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236634 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236637 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236641 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236645 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236648 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236650 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236653 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236655 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236658 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236660 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236662 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236665 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236667 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.243810 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236675 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236678 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236681 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236684 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236687 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236689 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236691 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236694 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236696 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236699 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236701 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236704 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236706 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236708 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236711 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236718 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236721 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236724 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236727 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.244305 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236729 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236732 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236734 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236736 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236739 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236741 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236743 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236746 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236748 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236750 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236753 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236756 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236758 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236760 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236763 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236765 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236767 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236770 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236772 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236775 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.244771 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236777 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236780 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236782 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236785 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236787 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236790 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236792 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236794 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236796 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236803 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236806 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236809 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236811 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236813 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236816 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236818 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236821 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236823 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236825 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.245278 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236828 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236830 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236833 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236835 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236838 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236840 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236843 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236846 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236848 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236850 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236853 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236856 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236858 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236861 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236863 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.236885 2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:10.246028 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236925 2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236929 2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236932 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236934 2578 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236937 2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236939 2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236942 2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236949 2578 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236952 2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236955 2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236957 2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236965 2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236968 2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236970 2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236973 2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236975 2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236977 2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236980 2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236982 2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236984 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:10.246761 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236987 2578 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236989 2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236991 2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236994 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236996 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.236999 2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237001 2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237003 2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237006 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237008 2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237010 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237013 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237015 2578 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237018 2578 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237020 2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237023 2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237025 2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237027 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237029 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237032 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:10.247429 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237039 2578 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237058 2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237061 2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237063 2578 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237066 2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237071 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237073 2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237076 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237078 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237081 2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237083 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237086 2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237088 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237091 2578 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237093 2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237095 2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237098 2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237101 2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237103 2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237105 2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:10.247923 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237108 2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237110 2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237113 2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237115 2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237118 2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237122 2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237126 2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237129 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237132 2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237134 2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237137 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237139 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237142 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237149 2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237152 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237155 2578 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237157 2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237162 2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237166 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:10.248469 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237169 2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237172 2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237174 2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237177 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237179 2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237182 2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237184 2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237186 2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237189 2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237191 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237194 2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237197 2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237199 2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237201 2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237204 2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237206 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237209 2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237213 2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237215 2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:10.248981 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237218 2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237220 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237223 2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237225 2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237228 2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237230 2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237233 2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:10.237235 2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.237296 2578 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.239620 2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.240597 2578 server.go:1014] "Starting client certificate rotation" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.240693 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.240749 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.243225 2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:10.249465 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.247604 2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:10.260247 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.260234 2578 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:31:10.264479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.264466 2578 log.go:25] "Validated CRI v1 image API" Oct 30 05:31:10.265427 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.265404 2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:31:10.271255 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.271234 2578 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d1a9131d-aff8-4c66-a2b9-045052cd2fc0:/dev/nvme0n1p4 f494dfe7-5134-460e-92d8-131f8dcff95a:/dev/nvme0n1p3] Oct 30 05:31:10.271319 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.271254 2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:31:10.277039 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.276933 2578 manager.go:217] Machine: {Timestamp:2026-10-30 05:31:10.275499651 +0000 UTC m=+0.376606113 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107679 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2965a21e37b59e6480273b2c6a3f13 SystemUUID:ec2965a2-1e37-b59e-6480-273b2c6a3f13 BootID:41f12e5e-cf39-4d48-8122-cb1e2a2a96ef Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:da:53:a9:8d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:da:53:a9:8d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:6a:bf:b9:66:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:31:10.277039 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.277038 2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:31:10.277173 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.277126 2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:31:10.280658 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.280633 2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:31:10.280803 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.280659 2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:31:10.281578 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.281568 2578 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:31:10.281618 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.281580 2578 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:31:10.281646 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.281617 2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:10.281646 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.281630 2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:10.282809 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.282799 2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:31:10.287276 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.287263 2578 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:31:10.287391 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.287382 2578 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:31:10.288208 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288199 2578 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:31:10.288245 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288212 2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:31:10.288245 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288222 2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:31:10.288245 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288231 2578 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:31:10.288245 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288239 2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:31:10.288649 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.288632 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:10.291114 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.291085 2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:31:10.299849 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.299831 2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:10.300751 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300738 2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:31:10.300801 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300770 2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:31:10.300949 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300937 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:31:10.300977 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300957 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:31:10.300977 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300964 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:31:10.300977 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300971 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:31:10.300977 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300977 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:31:10.301096 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.300985 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:31:10.301639 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301629 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:31:10.301674 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301642 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:31:10.301674 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301651 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:31:10.301674 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301658 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:31:10.301674 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301667 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:31:10.301790 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.301678 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:31:10.302431 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.302422 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:31:10.302461 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.302436 2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:31:10.306479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.306464 2578 server.go:1263] "Started kubelet" Oct 30 05:31:10.306623 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.306566 2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:31:10.306714 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.306555 2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:31:10.306714 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.306667 2578 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:31:10.307344 ip-10-0-136-35 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:31:10.308199 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.308150 2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:31:10.308652 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.308640 2578 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:31:10.314279 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.314261 2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:31:10.315340 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.315309 2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:31:10.316656 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.316640 2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:31:10.316768 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.316748 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.317435 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317419 2578 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:31:10.317435 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317438 2578 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:31:10.317555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317419 2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:31:10.317555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317531 2578 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:31:10.317555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317538 2578 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:31:10.317727 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317714 2578 factory.go:153] Registering CRI-O factory Oct 30 05:31:10.317763 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317733 2578 factory.go:223] Registration of the crio container factory successfully Oct 30 05:31:10.317793 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317783 2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:31:10.317821 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317794 2578 factory.go:55] Registering systemd factory Oct 30 05:31:10.317821 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317803 2578 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:31:10.317821 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317819 2578 factory.go:103] Registering Raw factory Oct 30 05:31:10.317901 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.317833 2578 manager.go:1199] Started watching for new ooms in manager Oct 30 05:31:10.318851 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.318839 2578 manager.go:319] Starting recovery of all containers Oct 30 05:31:10.332144 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.332128 2578 manager.go:324] Recovery completed Oct 30 05:31:10.335154 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.335130 2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-35.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:31:10.335154 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.335139 2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:31:10.336219 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.336206 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.336445 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.335136 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-35.ec2.internal.18e337a7c3ba7b48 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-35.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-35.ec2.internal,},FirstTimestamp:2026-10-30 05:31:10.306433864 +0000 UTC m=+0.407540326,LastTimestamp:2026-10-30 05:31:10.306433864 +0000 UTC m=+0.407540326,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-35.ec2.internal,}" Oct 30 05:31:10.337926 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.337908 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.337998 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.337944 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.337998 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.337956 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.338560 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.338543 2578 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:31:10.338560 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.338558 2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:31:10.338695 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.338574 2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:31:10.338695 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.338560 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.339231 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.339216 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.339298 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.339246 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.339298 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.339260 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.340795 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.340782 2578 policy_none.go:50] "Start" Oct 30 05:31:10.340849 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.340799 2578 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:31:10.340849 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.340812 2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:31:10.341034 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.341019 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:31:10.341133 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.341117 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-44lgs" Oct 30 05:31:10.342770 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.342760 2578 policy_none.go:44] "Start" Oct 30 05:31:10.349380 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.349356 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-44lgs" Oct 30 05:31:10.352300 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.352281 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:31:10.352355 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.352298 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:31:10.352355 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.352344 2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-35.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:31:10.352415 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.352349 2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-35.ec2.internal.18e337a7c59b0718 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-35.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-35.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-35.ec2.internal,},FirstTimestamp:2026-10-30 05:31:10.337926936 +0000 UTC m=+0.439033404,LastTimestamp:2026-10-30 05:31:10.337926936 +0000 UTC m=+0.439033404,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-35.ec2.internal,}" Oct 30 05:31:10.379849 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.379823 2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:31:10.382673 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.382659 2578 manager.go:341] "Starting Device Plugin manager" Oct 30 05:31:10.383326 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.383307 2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:31:10.383326 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383323 2578 server.go:87] "Starting device plugin registration server" Oct 30 05:31:10.383549 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383534 2578 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:31:10.383616 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383548 2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:31:10.383667 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383655 2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:31:10.383780 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383761 2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:31:10.383780 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.383774 2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:31:10.384261 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.384242 2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:31:10.384322 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.384293 2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.458344 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.458319 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:31:10.460192 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.459584 2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:31:10.460192 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.459609 2578 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:31:10.460192 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.459637 2578 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:31:10.460192 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.459685 2578 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:31:10.475700 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.475545 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:10.484640 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.484627 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.485440 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.485421 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.485522 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.485451 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.485522 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.485462 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.485522 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.485485 2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.506428 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.506410 2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.506494 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.506431 2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-35.ec2.internal\": node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.552159 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.552139 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.560069 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.560036 2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"] Oct 30 05:31:10.560125 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.560116 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.560918 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.560901 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.560985 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.560930 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.560985 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.560941 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.563195 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563182 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.563353 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563338 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.563422 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563393 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.563923 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563910 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.563986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563924 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.563986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563930 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.563986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563954 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.563986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563967 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.563986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.563969 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.566111 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.566099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.566152 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.566125 2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:10.566798 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.566783 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:10.566838 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.566815 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:10.566838 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.566832 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:10.592321 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.592305 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-35.ec2.internal\" not found" node="ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.594315 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.594301 2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-35.ec2.internal\" not found" node="ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.618913 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.618887 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.618913 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.618915 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.619065 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.618930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.652940 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.652909 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.719646 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.719619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.719743 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.719655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.719743 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.719689 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.720204 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.719675 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.720204 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.720162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.720453 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.720162 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.753636 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.753568 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.853751 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.853725 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:10.895694 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.895674 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.896794 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:10.896766 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:10.953836 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:10.953805 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.054464 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.054368 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.154856 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.154826 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.240496 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.240464 2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:31:11.240984 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.240602 2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:31:11.255274 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.255252 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.319261 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.319192 2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:11.355279 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.355245 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:26:11 +0000 UTC" deadline="2028-05-08 20:24:51.761266892 +0000 UTC" Oct 30 05:31:11.355279 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.355275 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13358h53m40.405994975s" Oct 30 05:31:11.355472 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.355302 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.455431 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.455397 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.490601 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:11.490570 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51317ae8aea2bacdb65b75f627da0c48.slice/crio-5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e WatchSource:0}: Error finding container 5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e: Status 404 returned error can't find the container with id 5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e Oct 30 05:31:11.491001 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:11.490976 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75abb82a1e3a44a283c8687c87965bd9.slice/crio-318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794 WatchSource:0}: Error finding container 318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794: Status 404 returned error can't find the container with id 318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794 Oct 30 05:31:11.496664 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.496648 2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:31:11.501819 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.501802 2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:11.556089 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.556061 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.656665 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:11.656601 2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:31:11.736277 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.736248 2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:11.817759 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.817730 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:31:11.844121 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.844098 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:11.845058 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.845027 2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:31:11.867274 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:11.867244 2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:12.289774 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.289735 2578 apiserver.go:52] "Watching apiserver" Oct 30 05:31:12.316010 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.315981 2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:31:12.321545 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.321510 2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:31:12.321971 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.321944 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-ttvn4","kube-system/global-pull-secret-syncer-f8w5w","kube-system/konnectivity-agent-gnmzd","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal","openshift-image-registry/node-ca-hh7pw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","openshift-multus/multus-additional-cni-plugins-98xfs","openshift-multus/network-metrics-daemon-6bv7l","openshift-ovn-kubernetes/ovnkube-node-n79z7","openshift-multus/multus-wl429","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq","openshift-cluster-node-tuning-operator/tuned-xprpl","openshift-dns/node-resolver-bv9np","openshift-network-diagnostics/network-check-target-kvzzn"] Oct 30 05:31:12.324943 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.324917 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wl429" Oct 30 05:31:12.327316 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.327297 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.330168 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.330146 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.331820 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.331780 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:12.331922 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.331859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:12.341907 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.341881 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.344060 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.344024 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.344060 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.344038 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.344216 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.344027 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:31:12.345289 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.345263 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.345393 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.345365 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:12.345527 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.345507 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vg6rp\"" Oct 30 05:31:12.345659 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.345642 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.346056 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.346013 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.346328 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.346311 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:31:12.346671 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.346651 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xpx6p\"" Oct 30 05:31:12.346766 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.346652 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2ztbt\"" Oct 30 05:31:12.346940 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.346923 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:31:12.347030 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.347015 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:31:12.349995 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.348436 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.351291 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.351271 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.353808 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.353785 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.353894 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.353857 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:12.356427 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.356397 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.359145 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.359119 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.362541 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.362125 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.367705 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.367578 2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:31:12.367705 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.367658 2578 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdst2" Oct 30 05:31:12.378577 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.378557 2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fdst2" Oct 30 05:31:12.418686 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.418666 2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:31:12.430115 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430079 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430320 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430131 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430320 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430219 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjtqk\" (UniqueName: \"kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.430320 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430264 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.430320 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430293 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430391 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430419 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wr2q\" (UniqueName: \"kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430476 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430533 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430565 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430600 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430593 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430620 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430648 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430724 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430765 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430791 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430824 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430851 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430878 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430905 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.430960 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430930 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430958 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.430992 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431019 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431102 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431163 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd7xg\" (UniqueName: \"kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vlw9\" (UniqueName: \"kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431218 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431251 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431284 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbj44\" (UniqueName: \"kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431357 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431384 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431448 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.431489 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431473 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431501 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431529 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431553 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431579 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431604 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431630 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431653 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431678 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431705 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqgxp\" (UniqueName: \"kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431841 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431874 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431902 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431929 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431964 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.431990 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432015 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.432304 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432041 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432068 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432094 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432140 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432165 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432192 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432217 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432245 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dnsf\" (UniqueName: \"kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432330 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432359 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432430 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432458 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dz89\" (UniqueName: \"kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432490 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432521 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432550 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432580 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432612 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432638 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432666 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxwr4\" (UniqueName: \"kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432693 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432723 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432749 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432778 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432806 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432833 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.433719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.432863 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.452156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452016 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:31:12.452156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452016 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:31:12.452156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452048 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.452156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452101 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.452156 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452131 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452201 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452279 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452282 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf7fl\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452286 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jtd4z\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452441 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:31:12.452555 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452459 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452636 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452651 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452658 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452674 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8j26\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452654 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452678 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452693 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452732 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjhr7\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452816 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452829 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:31:12.452843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.452838 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:31:12.453327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.453182 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:31:12.453327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.453279 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k22sc\"" Oct 30 05:31:12.453421 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.453350 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pqn7\"" Oct 30 05:31:12.464728 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.464675 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"75abb82a1e3a44a283c8687c87965bd9","Type":"ContainerStarted","Data":"318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794"} Oct 30 05:31:12.466048 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.466020 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerStarted","Data":"5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e"} Oct 30 05:31:12.533834 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533846 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533879 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533904 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533933 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533958 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533981 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533999 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.534028 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534007 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534042 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534091 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnsf\" (UniqueName: \"kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534125 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534134 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534154 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534180 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534216 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534243 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz89\" (UniqueName: \"kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534267 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534286 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534293 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534329 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534339 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534361 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534392 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534421 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.534486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534453 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwr4\" (UniqueName: \"kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534479 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534505 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534531 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534556 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534633 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534658 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534686 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534687 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.534708 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534714 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtqk\" (UniqueName: \"kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534749 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534759 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.533956 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534778 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534807 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.535327 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534813 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.534862 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535071 2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535262 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535323 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535364 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535564 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535693 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535766 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535780 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535849 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535879 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.535923 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.535994 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:13.035952846 +0000 UTC m=+3.137059315 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536020 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.536270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536022 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536062 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wr2q\" (UniqueName: \"kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536086 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536111 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536180 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536419 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536424 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536471 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536559 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536570 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536584 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.537137 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536609 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536620 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536611 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536657 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536660 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536668 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536697 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536712 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536724 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536750 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536753 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536776 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536804 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536830 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.536848 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536857 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536885 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7xg\" (UniqueName: \"kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.537959 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.536900 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:13.036882459 +0000 UTC m=+3.137988913 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.536969 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537019 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537067 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537105 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlw9\" (UniqueName: \"kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537112 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537157 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537185 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537211 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbj44\" (UniqueName: \"kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537236 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537261 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537280 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537287 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537312 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537340 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537374 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537399 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.538855 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537425 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537454 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537481 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537509 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537533 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537558 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537620 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgxp\" (UniqueName: \"kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537649 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537678 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537708 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537736 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537749 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537763 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537772 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537796 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.537896 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.541732 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538199 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538206 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538255 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538254 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538358 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538393 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538440 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538450 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538457 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538501 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538508 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538411 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538521 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538551 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538576 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538592 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.542433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538636 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538650 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.538861 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.539506 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.539654 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.540534 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.543088 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.541395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.565535 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.565506 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:12.565535 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.565540 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:12.565739 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.565555 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:12.565739 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:12.565663 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:13.065629172 +0000 UTC m=+3.166735639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:12.565739 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.565661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnsf\" (UniqueName: \"kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.566703 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.566287 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz89\" (UniqueName: \"kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.568519 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.568473 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wr2q\" (UniqueName: \"kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.569090 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.569043 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7xg\" (UniqueName: \"kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:31:12.569478 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.569455 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgxp\" (UniqueName: \"kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.570329 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.570309 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtqk\" (UniqueName: \"kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.573321 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.573274 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbj44\" (UniqueName: \"kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:12.573517 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.573493 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlw9\" (UniqueName: \"kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.573816 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.573794 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwr4\" (UniqueName: \"kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.580027 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.580008 2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:12.652871 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.652835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wl429" Oct 30 05:31:12.661519 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.661493 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-bv9np" Oct 30 05:31:12.672682 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.672502 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:12.674395 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.674257 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c19260_4322_43b7_8529_f7c520ad4feb.slice/crio-aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d WatchSource:0}: Error finding container aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d: Status 404 returned error can't find the container with id aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d Oct 30 05:31:12.680336 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.680151 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:31:12.690830 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.690797 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod199ee6eb_216b_471a_8195_b9a71b842c16.slice/crio-2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44 WatchSource:0}: Error finding container 2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44: Status 404 returned error can't find the container with id 2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44 Oct 30 05:31:12.711741 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.711704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:12.713828 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.713801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:31:12.723762 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.723732 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd20562d0_453c_476a_9556_d5ee0ec52041.slice/crio-09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f WatchSource:0}: Error finding container 09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f: Status 404 returned error can't find the container with id 09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f Oct 30 05:31:12.726107 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.726077 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod268fe774_ac5a_405e_8dc9_4c2f3ec9c993.slice/crio-8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83 WatchSource:0}: Error finding container 8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83: Status 404 returned error can't find the container with id 8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83 Oct 30 05:31:12.728878 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.728856 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:31:12.733004 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.732983 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:31:12.733208 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:12.733176 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:31:12.741746 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.741701 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aeae184_01f9_43a5_864d_e5ad89841760.slice/crio-aab1c38a7692dc490df45aa9fa11c11c2e1208fc2f347e4c66e998402a5de64d WatchSource:0}: Error finding container aab1c38a7692dc490df45aa9fa11c11c2e1208fc2f347e4c66e998402a5de64d: Status 404 returned error can't find the container with id aab1c38a7692dc490df45aa9fa11c11c2e1208fc2f347e4c66e998402a5de64d Oct 30 05:31:12.748042 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.748018 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19db8a82_fab8_4558_8fd0_727d660a23c5.slice/crio-0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c WatchSource:0}: Error finding container 0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c: Status 404 returned error can't find the container with id 0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c Oct 30 05:31:12.753160 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:31:12.753131 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f320485_3c0f_4555_947e_59bc38d66e7b.slice/crio-6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11 WatchSource:0}: Error finding container 6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11: Status 404 returned error can't find the container with id 6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11 Oct 30 05:31:13.043501 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.043444 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:13.043657 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.043528 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:13.043718 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.043668 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:13.043761 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.043731 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:14.043710259 +0000 UTC m=+4.144816750 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:13.043982 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.043954 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:13.044130 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.044027 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:14.044007508 +0000 UTC m=+4.145113978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:13.144981 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.144947 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:13.145301 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.145148 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:13.145301 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.145181 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:13.145301 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.145195 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:13.145301 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:13.145270 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:14.145248433 +0000 UTC m=+4.246354901 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:13.382435 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.382369 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:13 +0000 UTC" deadline="2028-07-19 03:32:05.971069056 +0000 UTC" Oct 30 05:31:13.382435 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.382435 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15070h0m52.588638624s" Oct 30 05:31:13.474292 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.474198 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gnmzd" event={"ID":"8320d815-b161-471e-97d6-dda571fef137","Type":"ContainerStarted","Data":"aa3b841a9c1aacfca3a67c34e7ebdd9c2d313133aa179c37aa80d980572f73ad"} Oct 30 05:31:13.479512 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.479474 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerStarted","Data":"6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11"} Oct 30 05:31:13.483970 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.483936 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xprpl" event={"ID":"4aeae184-01f9-43a5-864d-e5ad89841760","Type":"ContainerStarted","Data":"aab1c38a7692dc490df45aa9fa11c11c2e1208fc2f347e4c66e998402a5de64d"} Oct 30 05:31:13.489477 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.489448 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttvn4" event={"ID":"268fe774-ac5a-405e-8dc9-4c2f3ec9c993","Type":"ContainerStarted","Data":"8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83"} Oct 30 05:31:13.494179 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.494150 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bv9np" event={"ID":"e6c19260-4322-43b7-8529-f7c520ad4feb","Type":"ContainerStarted","Data":"aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d"} Oct 30 05:31:13.496729 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.496701 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wl429" event={"ID":"c8fad824-c375-4d0d-86aa-c1742b281b92","Type":"ContainerStarted","Data":"0a57efa7a885feae152388836df2dc751978ceef164b805527b49acaf8c9415d"} Oct 30 05:31:13.510368 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.510326 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"75abb82a1e3a44a283c8687c87965bd9","Type":"ContainerStarted","Data":"d80f23f796fd1d85d5a0676dfc4b882f5fa914fac5d094baf50e74494d67e75e"} Oct 30 05:31:13.515967 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.515750 2578 generic.go:356] "Generic (PLEG): container finished" podID="51317ae8aea2bacdb65b75f627da0c48" containerID="e2f1df8d09984d82ae14261ebf67210b51e1dd4a157eeeda5fbd1d8451fe8fcf" exitCode=0 Oct 30 05:31:13.515967 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.515849 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerDied","Data":"e2f1df8d09984d82ae14261ebf67210b51e1dd4a157eeeda5fbd1d8451fe8fcf"} Oct 30 05:31:13.520013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.519890 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" event={"ID":"19db8a82-fab8-4558-8fd0-727d660a23c5","Type":"ContainerStarted","Data":"0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c"} Oct 30 05:31:13.538499 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.538367 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f"} Oct 30 05:31:13.540443 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:13.540311 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hh7pw" event={"ID":"199ee6eb-216b-471a-8195-b9a71b842c16","Type":"ContainerStarted","Data":"2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44"} Oct 30 05:31:14.055217 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.055167 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:14.055396 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.055272 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:14.055458 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.055406 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:14.055510 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.055470 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:16.05545133 +0000 UTC m=+6.156557800 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:14.055889 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.055872 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:14.055948 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.055929 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:16.055913772 +0000 UTC m=+6.157020241 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:14.156184 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.155647 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:14.156184 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.155810 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:14.156184 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.155836 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:14.156184 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.155850 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:14.156184 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.155910 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:16.155888606 +0000 UTC m=+6.256995072 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:14.383286 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.383241 2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:13 +0000 UTC" deadline="2028-04-15 05:47:45.34929574 +0000 UTC" Oct 30 05:31:14.383286 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.383284 2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12792h16m30.966016681s" Oct 30 05:31:14.461367 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.461292 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:14.461367 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.461324 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:14.461583 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.461431 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:14.461583 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.461503 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:14.461694 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.461619 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:14.461750 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:14.461702 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:14.590894 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:14.590823 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" podStartSLOduration=2.590804619 podStartE2EDuration="2.590804619s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:13.553863656 +0000 UTC m=+3.654970132" watchObservedRunningTime="2026-10-30 05:31:14.590804619 +0000 UTC m=+4.691911110" Oct 30 05:31:15.562405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:15.561541 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerStarted","Data":"b1dc29fbd0b429ebc3938bf20aaef9d9846a01d6bf06d6509522c83ecdaa902b"} Oct 30 05:31:15.568799 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:15.568251 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerStarted","Data":"d3a992a0354cd6f10f7134df86102cf66bb2abc9b8f667ded3d3cebd4c423311"} Oct 30 05:31:15.661485 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:15.661421 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" podStartSLOduration=3.661402597 podStartE2EDuration="3.661402597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:15.610987851 +0000 UTC m=+5.712094326" watchObservedRunningTime="2026-10-30 05:31:15.661402597 +0000 UTC m=+5.762509072" Oct 30 05:31:16.073999 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.073956 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:16.074231 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.074057 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:16.074231 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.074217 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:16.074356 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.074286 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.074267261 +0000 UTC m=+10.175373716 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:16.074659 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.074548 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:16.074659 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.074616 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.074596684 +0000 UTC m=+10.175703151 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:16.174678 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.174612 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:16.174902 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.174850 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:16.174902 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.174879 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:16.174902 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.174892 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:16.175091 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.174956 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:20.174936743 +0000 UTC m=+10.276043215 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:16.462389 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.462312 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:16.462577 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.462454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:16.462920 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.462897 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:16.463072 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.463027 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:16.463151 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.463133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:16.463415 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:16.463223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:16.573740 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.573699 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="d3a992a0354cd6f10f7134df86102cf66bb2abc9b8f667ded3d3cebd4c423311" exitCode=0 Oct 30 05:31:16.573740 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:16.573745 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"d3a992a0354cd6f10f7134df86102cf66bb2abc9b8f667ded3d3cebd4c423311"} Oct 30 05:31:17.582384 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:17.581595 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gnmzd" event={"ID":"8320d815-b161-471e-97d6-dda571fef137","Type":"ContainerStarted","Data":"64be1fb1448325356a60b9d7d0f8c948cbd894bebc5772805b612dab800d1a2a"} Oct 30 05:31:17.626279 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:17.625358 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gnmzd" podStartSLOduration=2.700399375 podStartE2EDuration="6.625337752s" totalImagesPullingTime="3.924938377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:17.625314465 +0000 UTC m=+7.726420940" watchObservedRunningTime="2026-10-30 05:31:17.625337752 +0000 UTC m=+7.726444228" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:18.463190 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:18.463331 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:18.463801 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:18.463920 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:18.463994 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:18.464161 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:18.464084 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:18.588118 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:18.588036 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hh7pw" event={"ID":"199ee6eb-216b-471a-8195-b9a71b842c16","Type":"ContainerStarted","Data":"11538b80e01ae5db8f0f8fda4e47494f80cb55dde8bbb5b9e083f89cd992b7e7"} Oct 30 05:31:19.607482 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:19.607413 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" event={"ID":"19db8a82-fab8-4558-8fd0-727d660a23c5","Type":"ContainerStarted","Data":"ca5ad573700364dc4515aaea612d72b009116a0066bd51d1543d62c493fd7ea5"} Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.109607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.109698 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.109871 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.109946 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.10992536 +0000 UTC m=+18.211031832 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.110418 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:20.110528 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.110481 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.110462407 +0000 UTC m=+18.211568858 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:20.211064 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.210199 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:20.211064 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.210412 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:20.211064 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.210434 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:20.211064 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.210448 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:20.211064 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.210514 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.210494323 +0000 UTC m=+18.311600779 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:20.472417 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.472372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:20.472632 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.472590 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:20.473384 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.473361 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:20.473575 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.473545 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:20.473669 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.473645 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:20.473911 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:20.473831 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:20.550832 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.550804 2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:31:20.613425 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:20.613364 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" event={"ID":"19db8a82-fab8-4558-8fd0-727d660a23c5","Type":"ContainerStarted","Data":"11bce16ab1b78d73eb2ceaa6930225ea35b680f3db856de5919bfd98987ceaa1"} Oct 30 05:31:21.392292 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.392144 2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:31:20.550830437Z","UUID":"209d737f-3b4f-41db-b65d-fe6e7c79b7ec","Handler":null,"Name":"","Endpoint":""} Oct 30 05:31:21.393962 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.393934 2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:31:21.394089 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.393970 2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:31:21.617383 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.616717 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:21.617383 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.617185 2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:21.619958 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.619504 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerStarted","Data":"794a7ef99712d2746358009372e7df4146f3ec2f4d2b5fcc02994e12477caed8"} Oct 30 05:31:21.622014 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.621990 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-bv9np" event={"ID":"e6c19260-4322-43b7-8529-f7c520ad4feb","Type":"ContainerStarted","Data":"600a81359eb3d6904a8c117b386071a7c270724e3373e17aeed3af33f2cb1424"} Oct 30 05:31:21.624428 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.624399 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wl429" event={"ID":"c8fad824-c375-4d0d-86aa-c1742b281b92","Type":"ContainerStarted","Data":"b88eed94fb6d2b31b19fea8ee6f9c33a5cb7e79bea10796cdd7bc798e814b5ad"} Oct 30 05:31:21.660440 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.660369 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hh7pw" podStartSLOduration=5.577887961 podStartE2EDuration="10.660350252s" totalImagesPullingTime="5.082462291s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:18.632459204 +0000 UTC m=+8.733565680" watchObservedRunningTime="2026-10-30 05:31:21.660350252 +0000 UTC m=+11.761456727" Oct 30 05:31:21.706490 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.706427 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-wl429" podStartSLOduration=2.72117518 podStartE2EDuration="10.706412191s" totalImagesPullingTime="7.985237011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:21.705808428 +0000 UTC m=+11.806914903" watchObservedRunningTime="2026-10-30 05:31:21.706412191 +0000 UTC m=+11.807518666" Oct 30 05:31:21.749863 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:21.749773 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-bv9np" podStartSLOduration=2.437953201 podStartE2EDuration="10.74975316s" totalImagesPullingTime="8.311799959s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:21.749162832 +0000 UTC m=+11.850269339" watchObservedRunningTime="2026-10-30 05:31:21.74975316 +0000 UTC m=+11.850859635" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.460343 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:22.460709 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.461268 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.461303 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:22.461404 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:22.462667 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:22.461504 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:22.629556 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.629515 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" event={"ID":"19db8a82-fab8-4558-8fd0-727d660a23c5","Type":"ContainerStarted","Data":"2aae3f4efc4c6a30ab3baaae8a7427788d7704f1bd3c2eab391b00e9efd1fddf"} Oct 30 05:31:22.633213 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.632116 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="794a7ef99712d2746358009372e7df4146f3ec2f4d2b5fcc02994e12477caed8" exitCode=0 Oct 30 05:31:22.633213 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.632164 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"794a7ef99712d2746358009372e7df4146f3ec2f4d2b5fcc02994e12477caed8"} Oct 30 05:31:22.679645 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:22.679582 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" podStartSLOduration=3.144696768 podStartE2EDuration="11.679562806s" totalImagesPullingTime="8.534866038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:22.679047265 +0000 UTC m=+12.780153759" watchObservedRunningTime="2026-10-30 05:31:22.679562806 +0000 UTC m=+12.780669284" Oct 30 05:31:23.636271 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:23.636237 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-xprpl" event={"ID":"4aeae184-01f9-43a5-864d-e5ad89841760","Type":"ContainerStarted","Data":"be57b85eeee78c0faccfd127d43ac03b6a0cdab0c2751ea7e3ccd32795d03705"} Oct 30 05:31:23.637929 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:23.637903 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ttvn4" event={"ID":"268fe774-ac5a-405e-8dc9-4c2f3ec9c993","Type":"ContainerStarted","Data":"4824f413b2bf043c72e4ba9e4eaaf19850c8eb1ca0c838647320db6b9a141a10"} Oct 30 05:31:23.680104 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:23.680012 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-xprpl" podStartSLOduration=1.881487437 podStartE2EDuration="12.679998466s" totalImagesPullingTime="10.798511029s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:23.67915937 +0000 UTC m=+13.780265844" watchObservedRunningTime="2026-10-30 05:31:23.679998466 +0000 UTC m=+13.781104939" Oct 30 05:31:24.460769 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.460730 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:24.460945 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.460743 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:24.460945 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:24.460859 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:24.461081 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:24.460946 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:24.461081 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.460748 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:24.461197 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:24.461077 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:24.642831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.642794 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="700d346a219e7a9c0729b7986922cf86c6423371ac7dd4b18e5c97a8239b0422" exitCode=0 Oct 30 05:31:24.643449 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.642876 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"700d346a219e7a9c0729b7986922cf86c6423371ac7dd4b18e5c97a8239b0422"} Oct 30 05:31:24.714599 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:24.714498 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ttvn4" podStartSLOduration=5.284548406 podStartE2EDuration="13.714479654s" totalImagesPullingTime="8.429931248s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:23.717753374 +0000 UTC m=+13.818859842" watchObservedRunningTime="2026-10-30 05:31:24.714479654 +0000 UTC m=+14.815586128" Oct 30 05:31:25.648296 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:25.648208 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="1091e33ce1eef1bafb52400ad757f996ac7a50566abbb3216cda689eb5371f73" exitCode=0 Oct 30 05:31:25.648296 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:25.648271 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"1091e33ce1eef1bafb52400ad757f996ac7a50566abbb3216cda689eb5371f73"} Oct 30 05:31:26.462060 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:26.462013 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:26.462230 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:26.462138 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:26.462549 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:26.462530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:26.462657 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:26.462637 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:26.462711 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:26.462704 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:26.462773 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:26.462759 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:27.655381 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:27.655171 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="10118ab99bc6ce556b1ec52b99b23769ce2f5d1f2fc1ea3829f74d18b215153a" exitCode=0 Oct 30 05:31:27.655851 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:27.655516 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"10118ab99bc6ce556b1ec52b99b23769ce2f5d1f2fc1ea3829f74d18b215153a"} Oct 30 05:31:28.169498 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.169475 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:28.169582 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.169562 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:28.169617 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.169592 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:28.169655 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.169651 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:31:44.169633649 +0000 UTC m=+34.270740102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:28.169704 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.169668 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:28.169745 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.169721 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:44.169705557 +0000 UTC m=+34.270812009 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:28.270195 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.270122 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:28.270291 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.270274 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:28.270329 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.270296 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:28.270329 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.270305 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:28.270385 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.270351 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:31:44.270336564 +0000 UTC m=+34.371443031 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:28.460286 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.460246 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:28.460395 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.460374 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:28.460448 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.460269 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:28.460486 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.460454 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:28.460486 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.460471 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:28.460542 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:28.460518 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:28.660870 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.660632 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"1ec60079707e2097e544054891a7c029ac3c6dc18602ea0a17fee62a829b35e8"} Oct 30 05:31:28.661287 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.660868 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"f425c326669e5a9477e15d7cf25d5d36a9ba4d601e9961d41fbceea461fba828"} Oct 30 05:31:28.661287 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.660912 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"743075bc7378aacb919d8dfd0756639fe601467ccd054436c275db6041933599"} Oct 30 05:31:28.661287 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.660941 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"be5b162927a83d7f223aea39711831f59596c80990794f1ecfd2a35791f5aeb0"} Oct 30 05:31:28.661287 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.660974 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"5c82021669be61e37aba3c5e5b5099a31a133a11cb1a772fefaa624d6c78df20"} Oct 30 05:31:28.661287 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.661012 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"755fccae9b5be4d696e1d64f34a6530c144115cd986888b37985d4a7a27d2e1a"} Oct 30 05:31:28.663654 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.663630 2578 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="8a93516ac4fe4e0eaf09f81704dcd36604f330a953640bf5c675a6934d2bc590" exitCode=0 Oct 30 05:31:28.663771 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:28.663664 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerDied","Data":"8a93516ac4fe4e0eaf09f81704dcd36604f330a953640bf5c675a6934d2bc590"} Oct 30 05:31:29.668556 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:29.668518 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-98xfs" event={"ID":"3f320485-3c0f-4555-947e-59bc38d66e7b","Type":"ContainerStarted","Data":"a77fb856f4f3f176113601b7347ac0da676d938a071f2c919687a7f3d757a7ef"} Oct 30 05:31:29.740221 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:29.740173 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-98xfs" podStartSLOduration=8.164161888 podStartE2EDuration="18.740160789s" totalImagesPullingTime="10.575998901s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:29.739949785 +0000 UTC m=+19.841056256" watchObservedRunningTime="2026-10-30 05:31:29.740160789 +0000 UTC m=+19.841267271" Oct 30 05:31:30.462984 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:30.462951 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:30.463194 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:30.463056 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:30.463194 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:30.463077 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:30.463194 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:30.463136 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:30.463194 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:30.463175 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:30.463460 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:30.463288 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:30.674724 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:30.674685 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"eb20ce704b9e7f3fb330ecaaa5c76c93790a777bc4bf5b3e86e96096a35561e4"} Oct 30 05:31:32.460459 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.460431 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:32.460825 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.460482 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:32.460825 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.460485 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:32.460825 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:32.460550 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:32.460825 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:32.460671 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:32.460825 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:32.460755 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:32.683654 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.683618 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" event={"ID":"d20562d0-453c-476a-9556-d5ee0ec52041","Type":"ContainerStarted","Data":"b2fc2af5c830d97910c0d8e2a8885ecc0736ac16186470c4e92fcd4517d50746"} Oct 30 05:31:32.684112 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.684058 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:32.684204 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.684117 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:32.700178 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.700153 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:32.773645 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:32.773539 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" podStartSLOduration=6.3879890360000005 podStartE2EDuration="21.773526056s" totalImagesPullingTime="15.38553702s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:32.760504331 +0000 UTC m=+22.861610802" watchObservedRunningTime="2026-10-30 05:31:32.773526056 +0000 UTC m=+22.874632527" Oct 30 05:31:33.686509 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:33.686482 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:33.702329 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:33.702305 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:31:33.842203 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:33.842178 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv9np_e6c19260-4322-43b7-8529-f7c520ad4feb/dns-node-resolver/0.log" Oct 30 05:31:34.459954 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:34.459912 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:34.460125 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:34.459928 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:34.460125 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:34.460082 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:34.460224 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:34.459937 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:34.460224 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:34.460135 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:34.460292 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:34.460219 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:34.979566 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:34.979543 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hh7pw_199ee6eb-216b-471a-8195-b9a71b842c16/node-ca/0.log" Oct 30 05:31:35.237460 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:35.237383 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:35.237628 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:35.237498 2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:31:35.237995 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:35.237979 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:31:36.460072 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:36.460022 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:36.460072 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:36.460033 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:36.460072 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:36.460037 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:36.460567 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:36.460134 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:36.460567 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:36.460200 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:36.460567 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:36.460286 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:36.667687 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:36.667647 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:36.841954 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:36.841860 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:37.031101 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:37.031068 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:37.219026 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:37.218950 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:37.388358 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:37.388309 2578 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" limit=1024 Oct 30 05:31:38.460401 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:38.460358 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:38.460943 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:38.460429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:38.460943 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:38.460429 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:38.460943 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:38.460527 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:38.460943 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:38.460639 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:38.460943 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:38.460708 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:40.461340 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:40.461310 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:40.461768 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:40.461412 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:40.461768 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:40.461437 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:40.461768 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:40.461487 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:40.461768 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:40.461508 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:40.461768 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:40.461628 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:42.460565 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:42.460524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:42.461007 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:42.460524 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:42.461007 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:42.460627 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:42.461007 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:42.460537 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:42.461007 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:42.460747 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:42.461007 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:42.460786 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:44.176052 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.176023 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:44.176447 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.176083 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:44.176447 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.176160 2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:44.176447 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.176184 2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:44.176447 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.176215 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret podName:5690cc6f-94f7-4bc0-b271-35c91580a7e8 nodeName:}" failed. No retries permitted until 2026-10-30 05:32:16.176200123 +0000 UTC m=+66.277306597 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret") pod "global-pull-secret-syncer-f8w5w" (UID: "5690cc6f-94f7-4bc0-b271-35c91580a7e8") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:31:44.176447 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.176228 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs podName:016f2d58-bf54-4047-a2e0-8f14ee94825c nodeName:}" failed. No retries permitted until 2026-10-30 05:32:16.176221763 +0000 UTC m=+66.277328211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs") pod "network-metrics-daemon-6bv7l" (UID: "016f2d58-bf54-4047-a2e0-8f14ee94825c") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:31:44.276463 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.276436 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:44.276621 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.276562 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:31:44.276621 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.276580 2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:31:44.276621 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.276593 2578 projected.go:196] Error preparing data for projected volume kube-api-access-mmr6f for pod openshift-network-diagnostics/network-check-target-kvzzn: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:44.276717 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.276635 2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f podName:05b1f0ad-3ffd-4397-931e-60f5e9c69cc8 nodeName:}" failed. No retries permitted until 2026-10-30 05:32:16.276624664 +0000 UTC m=+66.377731127 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-mmr6f" (UniqueName: "kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f") pod "network-check-target-kvzzn" (UID: "05b1f0ad-3ffd-4397-931e-60f5e9c69cc8") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:31:44.460603 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.460530 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:44.460603 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.460545 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:44.460603 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:44.460547 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:44.460786 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.460629 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:44.460786 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.460737 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:44.460851 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:44.460816 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:46.460116 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:46.460077 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:46.460612 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:46.460133 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:46.460612 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:46.460148 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:46.460612 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:46.460259 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:46.460612 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:46.460297 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:46.460612 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:46.460345 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:48.460155 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:48.460130 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:48.460155 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:48.460162 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:48.460629 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:48.460135 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:48.460629 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:48.460223 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:48.460629 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:48.460324 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:48.460629 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:48.460386 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:50.461230 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:50.461196 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:50.461730 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:50.461327 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:50.461730 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:50.461372 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:50.461730 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:50.461427 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:50.461730 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:50.461508 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:50.461730 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:50.461595 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:52.460669 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:52.460638 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:52.460669 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:52.460667 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:52.461214 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:52.460649 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:52.461214 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:52.460761 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:52.461214 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:52.460846 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:52.461214 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:52.460931 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:54.460365 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.460336 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:54.460769 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.460391 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:54.460769 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.460402 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:54.460769 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.460499 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:54.460769 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.460526 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:54.460769 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.460599 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:54.647715 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.647679 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f8w5w"] Oct 30 05:31:54.647847 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.647758 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvzzn"] Oct 30 05:31:54.647847 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.647782 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6bv7l"] Oct 30 05:31:54.728519 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.728497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:54.728519 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.728514 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:54.728705 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:54.728496 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:54.728705 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.728601 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:54.728705 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.728681 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:54.728851 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:54.728742 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:55.126249 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:55.126224 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv9np_e6c19260-4322-43b7-8529-f7c520ad4feb/dns-node-resolver/0.log" Oct 30 05:31:56.460679 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:56.460652 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:56.460679 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:56.460677 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:56.461175 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:56.460746 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:56.461175 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:56.460655 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:56.461175 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:56.460833 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:56.461175 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:56.460909 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:56.524866 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:56.524838 2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hh7pw_199ee6eb-216b-471a-8195-b9a71b842c16/node-ca/0.log" Oct 30 05:31:58.460249 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:58.460217 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:31:58.460649 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:58.460260 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:31:58.460649 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:58.460316 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-kvzzn" podUID="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" Oct 30 05:31:58.460649 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:58.460383 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-f8w5w" podUID="5690cc6f-94f7-4bc0-b271-35c91580a7e8" Oct 30 05:31:58.460649 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:58.460426 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:31:58.460649 ip-10-0-136-35 kubenswrapper[2578]: E1030 05:31:58.460503 2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6bv7l" podUID="016f2d58-bf54-4047-a2e0-8f14ee94825c" Oct 30 05:31:59.691379 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.691347 2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeReady" Oct 30 05:31:59.691750 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.691445 2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:31:59.758972 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.758942 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4tpzp"] Oct 30 05:31:59.761768 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.761753 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-4xwsj"] Oct 30 05:31:59.761879 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.761864 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.764590 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.764570 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.785824 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.785799 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:31:59.785930 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.785846 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bmrwn\"" Oct 30 05:31:59.785930 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.785851 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7pqs8\"" Oct 30 05:31:59.785930 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.785889 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:31:59.786459 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.786447 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:31:59.786532 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.786509 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:31:59.786532 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.786521 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:31:59.786609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.786572 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:31:59.786609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.786598 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:31:59.798887 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.798867 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tpzp"] Oct 30 05:31:59.799401 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.799385 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xwsj"] Oct 30 05:31:59.879116 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.879094 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5pf44"] Oct 30 05:31:59.882860 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.882835 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5pf44" Oct 30 05:31:59.893165 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893146 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.893249 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893171 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.893249 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893202 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.893249 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893220 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vkmr\" (UniqueName: \"kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.893351 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893270 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc8b7\" (UniqueName: \"kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.893400 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893383 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.893433 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.893407 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.898207 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.898190 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:31:59.898495 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.898481 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvn8d\"" Oct 30 05:31:59.898551 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.898503 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:31:59.903686 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.903664 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5pf44"] Oct 30 05:31:59.994004 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.993979 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkmr\" (UniqueName: \"kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994101 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994009 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8b7\" (UniqueName: \"kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.994101 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994064 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:31:59.994170 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994117 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:31:59.994170 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994154 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87dv7\" (UniqueName: \"kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:31:59.994244 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994227 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994278 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994253 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.994278 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994272 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:31:59.994346 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994289 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994346 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994304 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994346 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994331 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994535 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994514 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994680 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994661 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.994843 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.994826 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:31:59.997998 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.997976 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:31:59.998089 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:31:59.997978 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:00.018942 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.018925 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkmr\" (UniqueName: \"kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:00.019423 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.019404 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8b7\" (UniqueName: \"kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:00.078594 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.078571 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:00.080640 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.080620 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:00.094882 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.094860 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.094968 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.094890 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.094968 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.094910 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87dv7\" (UniqueName: \"kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.094968 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.094960 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.095415 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.095395 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.095566 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.095547 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.097270 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.097248 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.121653 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.121624 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dv7\" (UniqueName: \"kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.194439 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.194401 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:00.422184 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.422124 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-4xwsj"] Oct 30 05:32:00.425141 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.425109 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tpzp"] Oct 30 05:32:00.425717 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:00.425694 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ae0fc4_ffe9_45fa_9367_3cb6307496ae.slice/crio-8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47 WatchSource:0}: Error finding container 8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47: Status 404 returned error can't find the container with id 8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47 Oct 30 05:32:00.439282 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:00.439261 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ecb3cba_e0f3_4342_bf46_da748d315f0b.slice/crio-e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe WatchSource:0}: Error finding container e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe: Status 404 returned error can't find the container with id e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe Oct 30 05:32:00.463810 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.463792 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:00.463924 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.463901 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:00.463924 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.463909 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:00.478926 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.478908 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:00.479070 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.478984 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:00.479070 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.479029 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:00.479070 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.479035 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:00.479217 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.479090 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:00.479217 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.479113 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:00.479217 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.479167 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:00.491189 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.491168 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5pf44"] Oct 30 05:32:00.494027 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:00.494007 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod151d83c7_e9e4_4e60_ad4b_591771415adc.slice/crio-98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3 WatchSource:0}: Error finding container 98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3: Status 404 returned error can't find the container with id 98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3 Oct 30 05:32:00.741450 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.741423 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pf44" event={"ID":"151d83c7-e9e4-4e60-ad4b-591771415adc","Type":"ContainerStarted","Data":"98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3"} Oct 30 05:32:00.742625 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.742602 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xwsj" event={"ID":"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae","Type":"ContainerStarted","Data":"8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47"} Oct 30 05:32:00.743872 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.743850 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tpzp" event={"ID":"5ecb3cba-e0f3-4342-bf46-da748d315f0b","Type":"ContainerStarted","Data":"992746a9a546c09f29a146e0bea136facaab6c28677ea74f1c786be137b6284d"} Oct 30 05:32:00.743954 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:00.743892 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tpzp" event={"ID":"5ecb3cba-e0f3-4342-bf46-da748d315f0b","Type":"ContainerStarted","Data":"e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe"} Oct 30 05:32:01.749351 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:01.749316 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tpzp" event={"ID":"5ecb3cba-e0f3-4342-bf46-da748d315f0b","Type":"ContainerStarted","Data":"1a33bc319fd832e1a59dd74faad7e6bc28f5c528670751aa4846aaf7f800ecc2"} Oct 30 05:32:02.359319 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.359278 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-nlj4r"] Oct 30 05:32:02.363945 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.363915 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.401289 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.401257 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:32:02.401510 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.401261 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:32:02.401510 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.401407 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lg7qg\"" Oct 30 05:32:02.439479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.439248 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:32:02.439479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.439307 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:32:02.439479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.439408 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:32:02.439479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.439446 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:32:02.514827 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514792 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514864 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514883 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514900 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hswx9\" (UniqueName: \"kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514918 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514934 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514962 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.514993 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.515013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.515011 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615459 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615517 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615546 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615611 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615619 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615613 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615719 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615753 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615779 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hswx9\" (UniqueName: \"kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.615831 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615814 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.616395 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.615842 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.616448 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.616409 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.616500 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.616427 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.617367 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.616653 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.617367 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.616788 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.619309 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.619258 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.619399 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.619330 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.642973 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.642898 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswx9\" (UniqueName: \"kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.687064 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.687006 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:02.711269 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:02.711233 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82812395_be27_46dd_8b16_bfab2c7fc22e.slice/crio-6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5 WatchSource:0}: Error finding container 6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5: Status 404 returned error can't find the container with id 6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5 Oct 30 05:32:02.754517 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.754481 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pf44" event={"ID":"151d83c7-e9e4-4e60-ad4b-591771415adc","Type":"ContainerStarted","Data":"0f1042150dc0db5cde1da23d40cf839f36fbe2177b6039b4c6f624bf7c3fd21e"} Oct 30 05:32:02.754938 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.754595 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5pf44" event={"ID":"151d83c7-e9e4-4e60-ad4b-591771415adc","Type":"ContainerStarted","Data":"d527727c6fb327421be82bd1667abeb19bb374a1239e6147927a63e5877ccf94"} Oct 30 05:32:02.754938 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.754630 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:02.756362 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.756335 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlj4r" event={"ID":"82812395-be27-46dd-8b16-bfab2c7fc22e","Type":"ContainerStarted","Data":"6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5"} Oct 30 05:32:02.799935 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:02.799880 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-5pf44" podStartSLOduration=1.948565537 podStartE2EDuration="3.799865228s" totalImagesPullingTime="1.851299691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:02.799100392 +0000 UTC m=+52.900206864" watchObservedRunningTime="2026-10-30 05:32:02.799865228 +0000 UTC m=+52.900971695" Oct 30 05:32:03.760664 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.760607 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-4xwsj" event={"ID":"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae","Type":"ContainerStarted","Data":"a496d29b07210f660bf3c1731ade03dbaa0b9b49cfed6f6bc77e9c54f1a2e867"} Oct 30 05:32:03.762278 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.762243 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlj4r" event={"ID":"82812395-be27-46dd-8b16-bfab2c7fc22e","Type":"ContainerStarted","Data":"d90d4a31ebac33ced284ff0808cc74304a1782caeb8e431a5ed4fa6b05b0558b"} Oct 30 05:32:03.764558 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.764524 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tpzp" event={"ID":"5ecb3cba-e0f3-4342-bf46-da748d315f0b","Type":"ContainerStarted","Data":"4951f3f860d2079d55cc09a13da6175012c2f2a6aaf495faf2d535031a11d66f"} Oct 30 05:32:03.801726 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.801671 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-4xwsj" podStartSLOduration=2.092558582 podStartE2EDuration="4.801657179s" totalImagesPullingTime="2.709098597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:03.800681701 +0000 UTC m=+53.901788173" watchObservedRunningTime="2026-10-30 05:32:03.801657179 +0000 UTC m=+53.902763647" Oct 30 05:32:03.805479 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.805444 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:03.809016 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.808987 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.824973 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.824932 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:32:03.825324 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.825232 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zsr6d\"" Oct 30 05:32:03.825324 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.825250 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:32:03.825324 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.825269 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:32:03.825324 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.825282 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:32:03.846632 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.843613 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:32:03.846632 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.843872 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:03.850999 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.850949 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4tpzp" podStartSLOduration=2.7368399610000003 podStartE2EDuration="4.850933851s" totalImagesPullingTime="2.11409389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:59 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:03.845887997 +0000 UTC m=+53.946994472" watchObservedRunningTime="2026-10-30 05:32:03.850933851 +0000 UTC m=+53.952040322" Oct 30 05:32:03.925596 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925562 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.925753 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925617 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.925753 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925646 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:32:03.925753 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925689 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.925753 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925747 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:32:03.925918 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925773 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.925918 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925787 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:32:03.925918 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925814 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.925918 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925869 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:32:03.926084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925933 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925976 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.925997 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926232 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.926080 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.926394 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926518 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.926452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926601 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.926583 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:03.926654 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:03.926623 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029014 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029191 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029225 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029301 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029351 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029420 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029460 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029526 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029582 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029617 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029655 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029722 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029759 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.031349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.029813 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.032414 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.032382 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.032815 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.032787 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.035446 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.035417 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.035542 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.035495 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.035542 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.035526 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.035947 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.035922 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.036031 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.035944 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.036144 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.036125 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.036210 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.036158 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.036340 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.036321 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.036423 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.036408 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.055152 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.055116 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.138707 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.138664 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:04.481491 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.481457 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:04.499565 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:04.499528 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5bb2921_d4ab_417d_ad12_1a0c57e3ef4f.slice/crio-3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167 WatchSource:0}: Error finding container 3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167: Status 404 returned error can't find the container with id 3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167 Oct 30 05:32:04.654741 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.654659 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7d56d56779-5qhql"] Oct 30 05:32:04.659336 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.659314 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.676776 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676743 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bs12it7als558\"" Oct 30 05:32:04.676776 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676743 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:32:04.676776 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676775 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:32:04.677084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676757 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:32:04.677084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676809 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:32:04.677084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676866 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:32:04.677084 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.676882 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-55j5k\"" Oct 30 05:32:04.684293 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.684255 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d56d56779-5qhql"] Oct 30 05:32:04.767986 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.767955 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167"} Oct 30 05:32:04.769612 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.769581 2578 generic.go:356] "Generic (PLEG): container finished" podID="82812395-be27-46dd-8b16-bfab2c7fc22e" containerID="d90d4a31ebac33ced284ff0808cc74304a1782caeb8e431a5ed4fa6b05b0558b" exitCode=0 Oct 30 05:32:04.769715 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.769683 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlj4r" event={"ID":"82812395-be27-46dd-8b16-bfab2c7fc22e","Type":"ContainerDied","Data":"d90d4a31ebac33ced284ff0808cc74304a1782caeb8e431a5ed4fa6b05b0558b"} Oct 30 05:32:04.834922 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.834895 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.834922 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.834926 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835140 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.834948 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835140 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835038 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835140 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835127 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835277 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835156 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835277 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835209 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835277 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835238 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.835277 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.835259 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-db858\" (UniqueName: \"kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936643 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936545 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936643 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936607 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936865 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936687 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936865 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936756 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936865 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936782 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db858\" (UniqueName: \"kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936865 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936839 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.936865 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936854 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.937116 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936874 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.937116 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.936899 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.938541 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.937778 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.938541 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.937867 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.940563 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.940530 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.940989 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.940968 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.941097 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.940993 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.941201 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.941171 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.942494 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.942468 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.945491 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.945465 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.961141 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.961108 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db858\" (UniqueName: \"kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:04.983062 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:04.983014 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:05.301192 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.301156 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7d56d56779-5qhql"] Oct 30 05:32:05.305111 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:05.305075 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b75558c_ebeb_4ae6_a623_676f3daef4f5.slice/crio-1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a WatchSource:0}: Error finding container 1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a: Status 404 returned error can't find the container with id 1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a Oct 30 05:32:05.701957 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.701881 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:05.775396 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.775360 2578 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb" exitCode=0 Oct 30 05:32:05.775790 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.775454 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb"} Oct 30 05:32:05.778848 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.778823 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlj4r" event={"ID":"82812395-be27-46dd-8b16-bfab2c7fc22e","Type":"ContainerStarted","Data":"2bee48900a7375c6d17236d3a80245b141dc4e9e37218d91b2efb0fdb63394d1"} Oct 30 05:32:05.778940 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.778870 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-nlj4r" event={"ID":"82812395-be27-46dd-8b16-bfab2c7fc22e","Type":"ContainerStarted","Data":"d1799e66e04e317bd02b7dff31b7cfbd4a1af5ead30f5643eeaedb8221bfce00"} Oct 30 05:32:05.780172 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.780153 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a"} Oct 30 05:32:05.878144 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:05.878078 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-nlj4r" podStartSLOduration=2.820408653 podStartE2EDuration="3.878056985s" totalImagesPullingTime="970.629238ms" totalInitContainerRuntime="87.019094ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:32:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:05.877464392 +0000 UTC m=+55.978570882" watchObservedRunningTime="2026-10-30 05:32:05.878056985 +0000 UTC m=+55.979163450" Oct 30 05:32:06.893923 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.893888 2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-689db8797f-sb5b8"] Oct 30 05:32:06.897899 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.897497 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.925462 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kbh7d\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.925503 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.925547 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.926214 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.926269 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.926350 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e6iddlm2f0hh2\"" Oct 30 05:32:06.927405 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.927333 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689db8797f-sb5b8"] Oct 30 05:32:06.955404 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955364 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955414 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxr4c\" (UniqueName: \"kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955452 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955496 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955536 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955557 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:06.955581 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:06.955575 2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056203 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056160 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056215 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr4c\" (UniqueName: \"kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056248 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056295 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056342 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056379 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.056419 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.056402 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.057409 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.057374 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.057955 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.057902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.058081 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.058061 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.059932 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.059902 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.060487 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.060459 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.061770 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.061744 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.087610 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.087543 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr4c\" (UniqueName: \"kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.212349 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.212011 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:07.540131 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.540075 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-689db8797f-sb5b8"] Oct 30 05:32:07.543904 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:07.543876 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977c05aa_1b1d_4280_bd13_a098a44bb0a7.slice/crio-82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15 WatchSource:0}: Error finding container 82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15: Status 404 returned error can't find the container with id 82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15 Oct 30 05:32:07.790421 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.790384 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24"} Oct 30 05:32:07.790609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.790456 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c"} Oct 30 05:32:07.790609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.790490 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89"} Oct 30 05:32:07.790609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.790522 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79"} Oct 30 05:32:07.790609 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.790546 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78"} Oct 30 05:32:07.791622 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.791596 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" event={"ID":"977c05aa-1b1d-4280-bd13-a098a44bb0a7","Type":"ContainerStarted","Data":"82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15"} Oct 30 05:32:07.793636 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.793616 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"0adb07131a7a9da0b98867e7243ba58782aa29b2cf6070037ce50c984056528d"} Oct 30 05:32:07.793719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.793664 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"d26b842f97d934897f7b6e6a3d25cccb0766a2a3f56a7e7ee50bcdbb02c7e03f"} Oct 30 05:32:07.793719 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:07.793688 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"e85095390afab0938a7f6c902a27dbc8dd9c0dec48a4096cbe21d345b1161741"} Oct 30 05:32:08.801802 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.801765 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerStarted","Data":"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651"} Oct 30 05:32:08.805406 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.805373 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"ab2f831050892e60b08401a58c88259d6cd87edc4e12e070c79572e6b5c6b83a"} Oct 30 05:32:08.805546 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.805451 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"0d1021989ea6533a42d82ac23cdf7add530eafb9099d1056519099470903ec56"} Oct 30 05:32:08.805546 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.805483 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" event={"ID":"7b75558c-ebeb-4ae6-a623-676f3daef4f5","Type":"ContainerStarted","Data":"e9c3b3419b8a85ed4101e4159d7fb2bbae9aabadd8542d08c2fdfe5e90786765"} Oct 30 05:32:08.805546 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.805530 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:08.893936 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:08.893794 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.620972577 podStartE2EDuration="5.893775118s" totalImagesPullingTime="3.259285283s" totalInitContainerRuntime="13.517258ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:03 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:08.880714479 +0000 UTC m=+58.981821015" watchObservedRunningTime="2026-10-30 05:32:08.893775118 +0000 UTC m=+58.994881590" Oct 30 05:32:09.810583 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:09.810536 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" event={"ID":"977c05aa-1b1d-4280-bd13-a098a44bb0a7","Type":"ContainerStarted","Data":"a2b2465cd8c390ef0459f7787125079615daba1031fc899f4f22695c8a24aec6"} Oct 30 05:32:09.855728 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:09.855673 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" podStartSLOduration=2.9487961499999997 podStartE2EDuration="5.855655787s" totalImagesPullingTime="2.906859637s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:04 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:08.936190558 +0000 UTC m=+59.037297029" watchObservedRunningTime="2026-10-30 05:32:09.855655787 +0000 UTC m=+59.956762259" Oct 30 05:32:09.855894 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:09.855858 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" podStartSLOduration=2.487625134 podStartE2EDuration="3.855851295s" totalImagesPullingTime="1.368226161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:09.855186681 +0000 UTC m=+59.956293152" watchObservedRunningTime="2026-10-30 05:32:09.855851295 +0000 UTC m=+59.956957767" Oct 30 05:32:12.767026 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:12.766992 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:14.817198 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:14.817175 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:16.237741 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.237694 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:16.238194 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.237783 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:16.252061 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.252023 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:16.260438 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.260413 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:16.264525 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.264501 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:16.270856 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.270830 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:16.338513 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.338480 2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:16.352764 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.352738 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:16.373004 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.372980 2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:16.381737 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.381707 2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:16.411811 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.411781 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:16.418123 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.418099 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:16.418749 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.418723 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:16.425634 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.425612 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:16.693265 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.693237 2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:16.699370 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.699335 2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:16.731398 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.731366 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-f8w5w"] Oct 30 05:32:16.738027 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:16.737938 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5690cc6f_94f7_4bc0_b271_35c91580a7e8.slice/crio-8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a WatchSource:0}: Error finding container 8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a: Status 404 returned error can't find the container with id 8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a Oct 30 05:32:16.748693 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.748658 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6bv7l"] Oct 30 05:32:16.751424 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:16.751396 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016f2d58_bf54_4047_a2e0_8f14ee94825c.slice/crio-0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b WatchSource:0}: Error finding container 0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b: Status 404 returned error can't find the container with id 0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b Oct 30 05:32:16.836010 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.835975 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bv7l" event={"ID":"016f2d58-bf54-4047-a2e0-8f14ee94825c","Type":"ContainerStarted","Data":"0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b"} Oct 30 05:32:16.837265 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:16.837242 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-f8w5w" event={"ID":"5690cc6f-94f7-4bc0-b271-35c91580a7e8","Type":"ContainerStarted","Data":"8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a"} Oct 30 05:32:17.008143 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:17.008037 2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-kvzzn"] Oct 30 05:32:17.011038 ip-10-0-136-35 kubenswrapper[2578]: W1030 05:32:17.011008 2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05b1f0ad_3ffd_4397_931e_60f5e9c69cc8.slice/crio-a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf WatchSource:0}: Error finding container a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf: Status 404 returned error can't find the container with id a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf Oct 30 05:32:17.849626 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:17.849596 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvzzn" event={"ID":"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8","Type":"ContainerStarted","Data":"a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf"} Oct 30 05:32:17.853386 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:17.853357 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bv7l" event={"ID":"016f2d58-bf54-4047-a2e0-8f14ee94825c","Type":"ContainerStarted","Data":"c47fdff8820ce3feada3b36f369be69d19a823dce18c80f7443439f25d237fc5"} Oct 30 05:32:18.860013 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:18.859961 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6bv7l" event={"ID":"016f2d58-bf54-4047-a2e0-8f14ee94825c","Type":"ContainerStarted","Data":"430f3d88b72dc9ef80a396363b13ed42fe6bcaac308ea98d503ebecaa23d9405"} Oct 30 05:32:18.903144 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:18.903077 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6bv7l" podStartSLOduration=66.931968131 podStartE2EDuration="1m7.903060792s" totalImagesPullingTime="971.092661ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:18.902127997 +0000 UTC m=+69.003234470" watchObservedRunningTime="2026-10-30 05:32:18.903060792 +0000 UTC m=+69.004167268" Oct 30 05:32:20.869846 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:20.869811 2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-kvzzn" event={"ID":"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8","Type":"ContainerStarted","Data":"8e3581e4e6febc47e037e78dff8820203875b8e20005036f0a022b24ff3b5c54"} Oct 30 05:32:20.870303 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:20.869922 2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:20.910808 ip-10-0-136-35 kubenswrapper[2578]: I1030 05:32:20.910762 2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-kvzzn" podStartSLOduration=66.791069631 podStartE2EDuration="1m9.910748993s" totalImagesPullingTime="3.119679362s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:20.910119034 +0000 UTC m=+71.011225507" watchObservedRunningTime="2026-10-30 05:32:20.910748993 +0000 UTC m=+71.011855464" Oct 30 05:32:21.489149 ip-10-0-136-35 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:32:21.500403 ip-10-0-136-35 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:32:21.500571 ip-10-0-136-35 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:32:21.500618 ip-10-0-136-35 systemd[1]: kubelet.service: Consumed 3.993s CPU time. Oct 30 05:32:21.514816 ip-10-0-136-35 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:32:21.588890 ip-10-0-136-35 kubenswrapper[6171]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:21.588890 ip-10-0-136-35 kubenswrapper[6171]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:32:21.588890 ip-10-0-136-35 kubenswrapper[6171]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:21.588890 ip-10-0-136-35 kubenswrapper[6171]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:32:21.590945 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590866 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.590945 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590945 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590950 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590953 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590956 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590959 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590961 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590965 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590968 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590971 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590973 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590976 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590978 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590981 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590984 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590986 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590989 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590992 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590994 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.590997 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591005 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.591017 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591008 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591012 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591014 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591017 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591020 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591023 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591025 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591028 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591031 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591033 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591036 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591038 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591060 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591062 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591065 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591068 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591070 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591073 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591082 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.591557 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591085 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591088 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591090 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591093 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591096 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591098 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591101 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591103 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591106 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591108 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591111 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591113 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591115 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591118 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591121 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591123 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591126 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591128 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591131 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591134 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.592059 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591136 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591138 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591141 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591143 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591146 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591148 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591151 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591153 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591155 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591158 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591160 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591164 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591169 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591175 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591179 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591182 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591184 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591187 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591189 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.592681 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591192 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591195 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591197 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591199 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591202 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591204 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591207 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591210 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591213 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591215 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591218 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591221 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591223 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591226 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591228 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591231 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591234 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591236 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591239 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591241 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.593188 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591244 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591246 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591249 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591252 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591256 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591260 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.591264 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592017 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592024 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592027 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592030 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592033 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592035 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592038 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592059 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592063 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592068 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592071 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592074 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.593713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592078 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592080 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592083 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592086 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592088 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592091 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592093 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592096 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592098 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592101 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592104 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592106 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592108 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592111 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592113 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592117 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592120 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592122 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592125 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592127 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.594225 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592131 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592133 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592136 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592138 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592141 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592143 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592146 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592148 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592150 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592153 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592155 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592158 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592161 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592163 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592166 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592168 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592171 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592173 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592176 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.594747 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592178 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592181 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592183 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592186 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592188 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592191 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592193 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592196 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592198 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592201 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592203 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592206 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592208 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592211 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592215 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592217 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592220 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592222 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592225 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592227 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.595251 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592231 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592235 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592238 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592241 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592244 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592246 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592249 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592251 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592254 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592256 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592259 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592261 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592263 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592266 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592269 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592271 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592273 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592276 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592278 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.595796 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592281 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592284 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592286 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592289 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592294 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592297 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592300 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592303 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592306 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592308 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592311 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592313 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592315 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592318 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592320 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592323 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592844 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592850 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592852 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.596303 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592855 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592858 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592861 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592863 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592866 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592869 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592871 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592874 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592877 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592880 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592882 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592885 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592887 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592890 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592893 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592895 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592898 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592900 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592903 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592906 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.596859 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592908 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592911 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592914 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592917 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592919 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592922 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592924 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592927 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592929 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592931 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592934 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592937 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592940 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592945 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592947 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592950 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592952 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592955 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592958 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592960 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.597413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592963 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592965 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592968 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592970 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592973 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592975 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592978 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592980 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592983 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592985 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592987 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592990 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592992 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592995 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.592998 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593000 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593002 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593005 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593007 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.597947 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593012 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593014 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593017 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593020 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593022 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593025 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593031 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593035 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593038 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593059 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593062 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593064 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593067 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593070 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593072 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593075 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593077 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593079 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593082 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.598789 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593084 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593087 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593089 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593092 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593094 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593097 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593099 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593102 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593104 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593107 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593110 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593113 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593115 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593119 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593122 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593124 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593127 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593129 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593132 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593135 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.599331 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593138 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593140 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593143 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593147 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593151 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593259 6171 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593281 6171 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593287 6171 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593291 6171 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593296 6171 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593299 6171 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593303 6171 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593307 6171 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593310 6171 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593313 6171 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593316 6171 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593319 6171 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593322 6171 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593325 6171 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593328 6171 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593331 6171 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593334 6171 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593337 6171 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:32:21.599843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593344 6171 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593346 6171 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593352 6171 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593355 6171 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593358 6171 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593362 6171 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593365 6171 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593368 6171 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593372 6171 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593375 6171 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593378 6171 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593381 6171 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593384 6171 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593387 6171 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593391 6171 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593394 6171 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593397 6171 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593400 6171 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593403 6171 flags.go:64] FLAG: --enable-server="true" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593406 6171 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593410 6171 flags.go:64] FLAG: --event-burst="100" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593413 6171 flags.go:64] FLAG: --event-qps="50" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593416 6171 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593419 6171 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:32:21.600452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593422 6171 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593426 6171 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593429 6171 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593432 6171 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593435 6171 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593438 6171 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593441 6171 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593444 6171 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593447 6171 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593450 6171 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593454 6171 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593457 6171 flags.go:64] FLAG: --feature-gates="" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593461 6171 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593464 6171 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593467 6171 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593470 6171 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593474 6171 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593477 6171 flags.go:64] FLAG: --help="false" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593481 6171 flags.go:64] FLAG: --hostname-override="ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593484 6171 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593486 6171 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593489 6171 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593493 6171 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:32:21.601079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593496 6171 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593499 6171 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593502 6171 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593505 6171 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593508 6171 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593511 6171 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593514 6171 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593516 6171 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593520 6171 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593523 6171 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593526 6171 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593529 6171 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593532 6171 flags.go:64] FLAG: --lock-file="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593535 6171 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593538 6171 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593541 6171 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593546 6171 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593549 6171 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593552 6171 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593556 6171 flags.go:64] FLAG: --logging-format="text" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593559 6171 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593562 6171 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593565 6171 flags.go:64] FLAG: --manifest-url="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593568 6171 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593572 6171 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:32:21.601678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593579 6171 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593583 6171 flags.go:64] FLAG: --max-pods="110" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593586 6171 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593589 6171 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593592 6171 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593595 6171 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593598 6171 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593601 6171 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593604 6171 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593611 6171 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593614 6171 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593617 6171 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593620 6171 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593623 6171 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593626 6171 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593628 6171 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593631 6171 flags.go:64] FLAG: --port="10250" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593634 6171 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593637 6171 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a8f6d2c986d55497" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593640 6171 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593643 6171 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593645 6171 flags.go:64] FLAG: --register-node="true" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593648 6171 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593652 6171 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593655 6171 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593658 6171 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:32:21.602323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593662 6171 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593666 6171 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593669 6171 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593672 6171 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593675 6171 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593678 6171 flags.go:64] FLAG: --runonce="false" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593682 6171 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593685 6171 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593688 6171 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593691 6171 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593693 6171 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593696 6171 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593700 6171 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593702 6171 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593705 6171 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593708 6171 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593711 6171 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593714 6171 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593717 6171 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593720 6171 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593723 6171 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593728 6171 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593731 6171 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593733 6171 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593738 6171 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:32:21.602966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593741 6171 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593743 6171 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593746 6171 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593749 6171 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593752 6171 flags.go:64] FLAG: --v="2" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593756 6171 flags.go:64] FLAG: --version="false" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593759 6171 flags.go:64] FLAG: --vmodule="" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593764 6171 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.593768 6171 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593931 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593936 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593939 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593942 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593945 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593949 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593951 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593954 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593957 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593959 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593962 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593964 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.603658 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593966 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593969 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593971 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593974 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593976 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593979 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593981 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593984 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593986 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593989 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593991 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593993 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593996 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.593998 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594001 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594003 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594005 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594008 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594011 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594014 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.604351 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594017 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594019 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594022 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594025 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594027 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594030 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594032 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594035 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594037 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594055 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594058 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594061 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594063 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594066 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594069 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594071 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594074 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594076 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594079 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594082 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.604944 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594084 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594087 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594089 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594092 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594094 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594096 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594100 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594104 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594107 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594109 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594112 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594114 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594117 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594120 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594123 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594125 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594128 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594131 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594135 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.605549 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594138 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594140 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594143 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594146 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594148 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594150 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594153 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594155 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594158 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594160 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594162 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594165 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594167 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594170 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594173 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594175 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594178 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594180 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594183 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.606114 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594185 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594188 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594191 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594193 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594196 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594198 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594201 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594203 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594208 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594211 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594214 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594219 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594221 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594224 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594226 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594229 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.606680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.594259 6171 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594302 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594306 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594309 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594312 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594315 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594317 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594320 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594322 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594325 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594328 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594330 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594333 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594335 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594339 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594342 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594344 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594347 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594349 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.607314 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594352 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594354 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594357 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594359 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594362 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594365 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594368 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594370 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594374 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594377 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594379 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594382 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594385 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594387 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594390 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594393 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594395 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594398 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594400 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.607803 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594403 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594405 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594408 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594410 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594414 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594417 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594419 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594421 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594425 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594428 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594430 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594433 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594435 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594438 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594440 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594442 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594445 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594447 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594450 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594452 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.608295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594455 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594459 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594462 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594464 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594467 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594469 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594471 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594474 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594477 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594479 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594481 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594484 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594486 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594489 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594491 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594493 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594496 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594498 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594501 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594503 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.608831 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594507 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594510 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594512 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594515 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594517 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594520 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594523 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594525 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594528 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594530 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594533 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594535 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594538 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594542 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594544 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594547 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594549 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594553 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594557 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.609383 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594560 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594562 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594565 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594567 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594570 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594572 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594575 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594577 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594580 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.594584 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.599235 6171 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.599252 6171 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.599270 6171 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.599275 6171 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599346 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599351 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599355 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599358 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599360 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.609864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599363 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599366 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599369 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599371 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599374 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599376 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599379 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599382 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599384 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599387 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599389 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599392 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599394 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599397 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599401 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599405 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599409 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599412 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599414 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599416 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.610395 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599419 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599421 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599425 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599427 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599430 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599432 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599435 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599437 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599440 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599442 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599445 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599448 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599451 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599453 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599456 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599458 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599461 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599464 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599466 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.610908 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599469 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599471 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599476 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599478 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599481 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599484 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599486 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599489 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599492 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599494 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599496 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599499 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599502 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599504 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599506 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599509 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599512 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599514 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599517 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.611420 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599521 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599536 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599540 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599543 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599546 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599548 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599551 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599553 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599556 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599558 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599561 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599564 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599566 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599569 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599571 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599574 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599577 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599580 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599582 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599585 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.611900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599587 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599590 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599593 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599595 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599597 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599600 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599602 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599605 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599607 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599609 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599612 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599615 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599617 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599620 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599622 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599625 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599627 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599630 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599632 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.612542 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599635 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599637 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599640 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599643 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.599672 6171 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599819 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599824 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599827 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599830 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599833 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599837 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599840 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599842 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599845 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599848 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599854 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.613368 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599857 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599859 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599862 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599864 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599868 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599870 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599873 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599875 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599878 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599880 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599883 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599886 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599888 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599891 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599893 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599896 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599898 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599901 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599903 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599906 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.613977 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599908 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599911 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599913 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599915 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599919 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599923 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599926 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599929 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599933 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599937 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599940 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599944 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599947 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599949 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599952 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599954 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599957 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599959 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.614941 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599961 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599964 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599967 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599969 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599972 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599974 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599977 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599980 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599982 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599985 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599987 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599990 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599992 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599994 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599997 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.599999 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600002 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600004 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600006 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600009 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.615905 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600011 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600014 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600017 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600020 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600022 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600028 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600030 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600033 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600035 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600038 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600059 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600063 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600067 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600072 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600074 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600077 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600080 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600082 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600085 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600088 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.616914 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600090 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600093 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600095 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600098 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600100 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600103 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600105 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600108 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600110 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600113 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600115 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600118 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600120 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600122 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600125 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600129 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.617801 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600132 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.600157 6171 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600190 6171 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600193 6171 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600196 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600199 6171 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600201 6171 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600204 6171 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600208 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600211 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600214 6171 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600216 6171 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600219 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600221 6171 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600224 6171 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600226 6171 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:32:21.618523 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600229 6171 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600231 6171 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600234 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600237 6171 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600239 6171 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600242 6171 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600244 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600247 6171 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600249 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600251 6171 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600254 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600257 6171 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600260 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600262 6171 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600265 6171 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600267 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600270 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600273 6171 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600277 6171 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:32:21.619302 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600279 6171 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600282 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600284 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600287 6171 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600289 6171 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600293 6171 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600295 6171 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600298 6171 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600300 6171 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600303 6171 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600306 6171 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600308 6171 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600311 6171 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600313 6171 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600315 6171 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600318 6171 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600320 6171 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600323 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600325 6171 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600328 6171 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:32:21.620109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600330 6171 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600332 6171 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600335 6171 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600337 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600340 6171 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600342 6171 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600346 6171 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600348 6171 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600350 6171 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600353 6171 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600356 6171 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600366 6171 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600370 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600373 6171 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600375 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600378 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600380 6171 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600385 6171 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600387 6171 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600390 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:32:21.620972 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600392 6171 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600395 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600397 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600400 6171 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600402 6171 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600405 6171 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600407 6171 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600410 6171 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600412 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600415 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600417 6171 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600420 6171 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600422 6171 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600425 6171 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600427 6171 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600430 6171 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600433 6171 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600435 6171 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600437 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600440 6171 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:32:21.621576 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600442 6171 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600445 6171 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600449 6171 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600454 6171 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600458 6171 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600461 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600466 6171 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600470 6171 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600474 6171 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600477 6171 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600480 6171 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600482 6171 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:21.600485 6171 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.600536 6171 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601485 6171 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601538 6171 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601710 6171 server.go:1014] "Starting client certificate rotation" Oct 30 05:32:21.622163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601851 6171 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601903 6171 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:26:11 +0000 UTC" deadline="2028-05-10 00:14:08.965095457 +0000 UTC" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.601931 6171 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13386h41m47.363167782s" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.602163 6171 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.603151 6171 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.604288 6171 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.605947 6171 log.go:25] "Validated CRI v1 image API" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.606642 6171 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:32:21.622633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.608547 6171 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d1a9131d-aff8-4c66-a2b9-045052cd2fc0:/dev/nvme0n1p4 f494dfe7-5134-460e-92d8-131f8dcff95a:/dev/nvme0n1p3] Oct 30 05:32:21.622877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.608568 6171 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a57efa7a885feae152388836df2dc751978ceef164b805527b49acaf8c9415d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a57efa7a885feae152388836df2dc751978ceef164b805527b49acaf8c9415d/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a/userdata/shm major:0 minor:288 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83/userdata/shm major:0 minor:87 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47/userdata/shm major:0 minor:204 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a/userdata/shm major:0 minor:371 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf/userdata/shm major:0 minor:375 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aa3b841a9c1aacfca3a67c34e7ebdd9c2d313133aa179c37aa80d980572f73ad/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aa3b841a9c1aacfca3a67c34e7ebdd9c2d313133aa179c37aa80d980572f73ad/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~projected/kube-api-access-bbj44:{mountpoint:/var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~projected/kube-api-access-bbj44 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~secret/metrics-certs major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8/volumes/kubernetes.io~projected/kube-api-access-mmr6f:{mountpoint:/var/lib/kubelet/pods/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8/volumes/kubernetes.io~projected/kube-api-access-mmr6f major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~projected/kube-api-access-87dv7:{mountpoint:/var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~projected/kube-api-access-87dv7 major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~secret/metrics-tls major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/199ee6eb-216b-471a-8195-b9a71b842c16/volumes/kubernetes.io~projected/kube-api-access-9vlw9:{mountpoint:/var/lib/kubelet/pods/199ee6eb-216b-471a-8195-b9a71b842c16/volumes/kubernetes.io~projected/kube-api-access-9vlw9 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/19db8a82-fab8-4558-8fd0-727d660a23c5/volumes/kubernetes.io~projected/kube-api-access-sxwr4:{mountpoint:/var/lib/kubelet/pods/19db8a82-fab8-4558-8fd0-727d660a23c5/volumes/kubernetes.io~projected/kube-api-access-sxwr4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/268fe774-ac5a-405e-8dc9-4c2f3ec9c993/volumes/kubernetes.io~projected/kube-api-access-qqgxp:{mountpoint:/var/lib/kubelet/pods/268fe774-ac5a-405e-8dc9-4c2f3ec9c993/volumes/kubernetes.io~projected/kube-api-access-qqgxp major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3f320485-3c0f-4555-947e-59bc38d66e7b/volumes/kubernetes.io~projected/kube-api-access-5dnsf:{mountpoint:/var/lib/kubelet/pods/3f320485-3c0f-4555-947e-59bc38d66e7b/volumes/kubernetes.io~projected/kube-api-access-5dnsf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~projected/kube-api-access-6wr2q:{mountpoint:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~projected/kube-api-access-6wr2q major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5690cc6f-94f7-4bc0-b271-35c91580a7e8/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/5690cc6f-94f7-4bc0-b271-35c91580a7e8/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~projected/kube-api-access-8vkmr:{mountpoint:/var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~projected/kube-api-access-8vkmr major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~projected/kube-api-access-db858:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~projected/kube-api-access-db858 major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~projected/kube-api-access-hswx9:{mountpoint:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~projected/kube-api-access-hswx9 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8320d815-b161-471e-97d6-dda571fef137/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/8320d815-b161-471e-97d6-dda571fef137/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~projected/kube-api-access-mxr4c:{mountpoint:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~projected/kube-api-access-mxr4c major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/alertmanager/9 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/config-reloader/6 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~empty-dir/config-out major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/kube-api-access-56cpm:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/kube-api-access-56cpm major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/tls-assets major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/config-volume major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/web-config major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~projected/kube-api-access-mc8b7:{mountpoint:/var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~projected/kube-api-access-mc8b7 major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~secret/cert major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8fad824-c375-4d0d-86aa-c1742b281b92/volumes/kubernetes.io~projected/kube-api-access-hd7xg:{mountpoint:/var/lib/kubelet/pods/c8fad824-c375-4d0d-86aa-c1742b281b92/volumes/kubernetes.io~projected/kube-api-access-hd7xg major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~projected/kube-api-access-5dz89:{mountpoint:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~projected/kube-api-access-5dz89 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e6c19260-4322-43b7-8529-f7c520ad4feb/volumes/kubernetes.io~projected/kube-api-access-fjtqk:{mountpoint:/var/lib/kubelet/pods/e6c19260-4322-43b7-8529-f7c520ad4feb/volumes/kubernetes.io~projected/kube-api-access-fjtqk major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f2c8d530aad6054e9795d1ae19e4c127d9cb309505165e5eaeb697092d2a22b8/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/c4a1050efc5862542293250b0b56f72c519f691434dd68ff6ebb2faac140967e/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/ea66f1d4b61487c56c86303457476b6243a8bb83284895d8eaedf1c5a77e1d55/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/2db7ac36e14f8b05f3a14f9119bcabbb6c460a22e309ae669230fc9c42d91932/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/dbde09beb551ef34a50e93fa75736ee77a109e1a8fc6438420801393c0fd0b5a/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/841edcdcc703fa553b30ae07ab8f85291d351d0b02ba5b40a421e7697218179a/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/846c4752839be2c3a5a42c026dcdb31bf288034edc5cbf69e35967f7aa1412a7/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/2be9c9c6018b9437da554b3f34c5cefd1919fecfc671e7dc391d271f09bc0899/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/34473f367a53ccc34af3960db69b008d7dd02a9e30f10e5551a7841025a42015/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/c38409a3dd912f9d4e1393b5c9f7cf4a0e1ea7e2ac2884d9e2a83dfac53287df/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/28e41a3500e9c990af46cbac0aaa6f4557afc5dae0ee0489cc587b0b377632cd/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/23d08a684a7534802d9d55664e39b51fdf49457fc8b34eec96cb6ba952cbff31/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/319b2a11f017c490029fa953af9c75e7f1f8498722a63443e5280edf9f5c0865/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/74adba6ff92eeb44ecc12378a4929eb9e1527d6d9506cf913a8763e198bae2d1/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/6ecdcc3fc2e7fc8e6f16e23fc9570c3e98511409ab83bc114d32cec9fd8c623b/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/bd5f7e1e971dcd42b9bca3c0051e21e32565358a8fe089c8d4dab0f930529872/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/62194e98b94b0bf56f6348dbb6f5464dee27ef61ea6a31162152d2c9162460ec/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/2d384825bea401ad3fd1a5d8ed9e19429ecea4139b131fde107179602c852c78/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/29c934c6add4b501182f2b00394242d9b43a9e92aa33b0726a96d2b4dc3f0ace/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/fb7e3fe48e981ca07d6f24f5e755bf33a19b91fde387774c2e9399b85b46255f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/de11ba99797c43c9d3ca2d359ec6b748d7b7cec03038d9843df1181a6b726dc6/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/b5eceef981bd1e4163f008bb2b3df915bb2fef7d10c9875484e9b510ea109b2e/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/50c7e3d57a7b70df2c4b181292c4d85d30d6e63e48c3237f523168be09f36cb5/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3fc44192c5daa903b5a68fc1a2341541e87168b346d0d25da705c5896f5c00ed/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/37ce3a68b9a7a96c33a7c37491151d395f027901b5e140ca14fca2c95d3e28bc/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/ee9a87a999beaad1451c2999a8828294b3c69d42f4f03db501968e11c9fcd73c/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/37ff45b755b682426aa110d7fb1628e2421fef7fb2cd05970e53c1891c14623f/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/2c5cd945161b68f84527d9cd1342bcea13478053cadfa9c2fe35f2c0c801e81c/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-244:{mountpoint:/var/lib/containers/storage/overlay/93083c338e0985b700a9e60c32ed29048d2fa31310d12f7c9ff213c0576d94f9/merged major:0 minor:244 fsType:overlay blockSize:0} overlay_0-246:{mountpoint:/var/lib/containers/storage/overlay/9fd69f1d65a7eb95cc8ff2fdc840dbef66d95e1e141eebd46fa16d81e0df2077/merged major:0 minor:246 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/549df2f7079458bc08f4bb9560e90a35895c6f0b2fe7bba00ec7e84af45fae69/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/d6af6fc1eff6aa0bf229f57ad8a7fd0650fa5f272a4c7b5b77e3ed0e39ec7e32/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/3b616db0b2491aa54c7ae5193eaf18683bf57b2cb1c60cfe4308aa72a581eb20/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/aee6f4edd7e01f68f4a2de844fdb8ce77272c343105c213b3ad8c0b91b93d030/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-290:{mountpoint:/var/lib/containers/storage/overlay/a5335bbc774fc88a72e87fd4f6d73932d27949b1c07a12a24eb6a1f7cac4f441/merged major:0 minor:290 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/969cc2c6a8400868044d8289480421a392e684f33d4740bd507f7f99ef443cc4/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/46253608cb6238f86b5aef5186a3454ebccee9bebf181be7cc1c71163a0d5f13/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/3f227da0a513c492f0430f36bfb55c981bcd6f3f77f2aebb98cf9258101c2a51/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/743735edf8f450da128261e149ff9d4587a067cc3edc0383b3800b0be8dfe446/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-314:{mountpoint:/var/lib/containers/storage/overlay/8730ceb95ecd23fe77bb255f8ed64c4dbd2c9a4187a56632cbca2e693540828b/merged major:0 minor:314 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/0025e9c4da460c27399b1845a5d7f8e2895b049672d211658136ad93006d386f/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-324:{mountpoint:/var/lib/containers/storage/overlay/5e75288a38d60cefaa018a5d051f9d98b0b0c8cdcf0dc5e0a2609374c80c14a5/merged major:0 minor:324 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/6a4696776e5205838b2b9c7d7836577f49da8c13ed2340765793bc800fe8d010/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/abad9d2a2cafde52d558a9be4d2fd9d61eab78fba97ada71f4c19164d54f72c0/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/5773da49daa7ae14377fa735bd1b32c0125c50544c7a37941d5ebead3ad28f2f/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/4d99ceb43dfa8e000cd1e21668c723aba7c831929729c9d12467019fb232415e/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/b0cfbecc94bc3d1a4e2dc1c6a4237b80a9fdf58cb41a94515577f7e2dd7dba28/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-353:{mountpoint:/var/lib/containers/storage/overlay/355904c6ecdeada2095cff1491f6cbb8ecf15037f47c6c2ce9811a55c4c08510/merged major:0 minor:353 fsType:overlay blockSize:0} overlay_0-359:{mountpoint:/var/lib/containers/storage/overlay/6dac7c79791c54e81299048ac44814123b0adaca11df9e45b29314b444ddf437/merged major:0 minor:359 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/d2161a65675d07bc20d75181aef48b829e01a276ef98a6ae5dc80b7bc7d152cd/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/c317ea4a6c15e3e26ade2e9d81fa9edf1d7a81eb263d1c074603bd3891bb26e2/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/5e963062b02bad6fce0ad8f85c26363b28c2b934f04edfeeb2bfb27e7f1098c3/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-383:{mountpoint:/var/lib/containers/storage/overlay/12676c9c126a250f21c6c8b7c1931279e5f2c80cfe354bae8638f6f6826a8987/merged major:0 minor:383 fsType:overlay blockSize:0} overlay_0-385:{mountpoint:/var/lib/containers/storage/overlay/ea8c0f0fb01b7c8565e70ce419f806249c514b21d0d0f30c451e7389bc09fb80/merged major:0 minor:385 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/9d166da94b182d47bc8a1b1a0e11d3a4ad8019d93d7d5d97daf47960b4b2852a/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/9cc9f5107aeea99f039bdccda295bfa03c42750a89231db0f70be80176dd3531/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/5216a55398600f14f52deacf966c6c9559ba31fd3da84d154c70fe6567ef4f95/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/602a421dc0d4943d998fbeca2ed1749fbe4bf8625667b68fb2522a4dbf3c11de/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/5e9f22f5c1f5a6248187afbbe1b5ea6b054bf7f4bb467db946c4de3d55d43777/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a6704d020eff27d45a456e489807875b39e866c2730dc2e64308a547b88b5ec3/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/5cf940ca4f903e4dad2ed86a66c99a3352d479258ed0402730476928d70058c3/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/c1ca01610455c8e61422dcaa1aebe14259f29b50195889d4de19e92c7883a992/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/08e3a68d0c434c7e19d6f4e9ba5cf66211a030a634c1d9f7c4c533e3e9918cab/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/ed2e32099bcf88d4692227df508ba94ebed6c6e7ca4b2a8455d9ff6ac47fa875/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-93:{mountpoint:/var/lib/containers/storage/overlay/d4f4a3dadad0d95a14fb7c5be4dff319bbc1d9d15135b5979099d0a90f288af3/merged major:0 minor:93 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/a28058785402c5f50082984bc2852a8b5e80a9f1281c7116de9eae160bd15250/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.617975 6171 manager.go:217] Machine: {Timestamp:2026-10-30 05:32:21.617166183 +0000 UTC m=+0.057543368 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099970 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2965a21e37b59e6480273b2c6a3f13 SystemUUID:ec2965a2-1e37-b59e-6480-273b2c6a3f13 BootID:41f12e5e-cf39-4d48-8122-cb1e2a2a96ef Filesystems:[{Device:/run/containers/storage/overlay-containers/0051983c9a8ddaf31ba3b49791f25f8c3d15004551f81bf0abb1b5ed9cdebe1c/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e05aea47c7faba1d9b3b2eafff0eefdfe11d1fe44ce39c90abb3ab14b07759fe/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/98f4b4dcfb7c96834985910bc21cab90780802cc9ef9bb41d14b7228701450b3/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-359 DeviceMajor:0 DeviceMinor:359 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9c44857ca94da641acc963db2e877cb96de5f00207093d447f95558ca5f2acf/userdata/shm DeviceMajor:0 DeviceMinor:375 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aa3b841a9c1aacfca3a67c34e7ebdd9c2d313133aa179c37aa80d980572f73ad/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-385 DeviceMajor:0 DeviceMinor:385 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-93 DeviceMajor:0 DeviceMinor:93 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/kube-api-access-56cpm DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8/volumes/kubernetes.io~projected/kube-api-access-mmr6f DeviceMajor:0 DeviceMinor:370 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~projected/kube-api-access-87dv7 DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8efc684afcc702a9093cc79790861c591f63b99eb98c1304e7f7361b9f97d42a/userdata/shm DeviceMajor:0 DeviceMinor:371 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d1816577c01eb79e9ce046c06a543f798e57c746b6cae16eb19700f9e643e83/userdata/shm DeviceMajor:0 DeviceMinor:87 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-290 DeviceMajor:0 DeviceMinor:290 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2bcafd242ddbc1e9d3e0eb9e8f64fdc1f108865893b76d8b8934ca2d43d70d44/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-353 DeviceMajor:0 DeviceMinor:353 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/19db8a82-fab8-4558-8fd0-727d660a23c5/volumes/kubernetes.io~projected/kube-api-access-sxwr4 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e6c19260-4322-43b7-8529-f7c520ad4feb/volumes/kubernetes.io~projected/kube-api-access-fjtqk DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:364 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/016f2d58-bf54-4047-a2e0-8f14ee94825c/volumes/kubernetes.io~projected/kube-api-access-bbj44 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-246 DeviceMajor:0 DeviceMinor:246 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b82444d9baec3ba87a31d8f7e57f85f8678ea5cfb3895908ff5c28de74f441a/userdata/shm DeviceMajor:0 DeviceMinor:288 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~projected/kube-api-access-8vkmr DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:260 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8e956ad15a1003fa24d75047e7fad3cc10be34dd818b1ae8b9808c4b8323ab47/userdata/shm DeviceMajor:0 DeviceMinor:204 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:293 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~projected/kube-api-access-mxr4c DeviceMajor:0 DeviceMinor:295 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5690cc6f-94f7-4bc0-b271-35c91580a7e8/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:369 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/5ecb3cba-e0f3-4342-bf46-da748d315f0b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~projected/kube-api-access-hswx9 DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/151d83c7-e9e4-4e60-ad4b-591771415adc/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae/volumes/kubernetes.io~projected/kube-api-access-mc8b7 DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6615ea7e4f88649e815fd63625fee1b8973ef24787ef68a18a213afa1da66fa5/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a57efa7a885feae152388836df2dc751978ceef164b805527b49acaf8c9415d/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d20562d0-453c-476a-9556-d5ee0ec52041/volumes/kubernetes.io~projected/kube-api-access-5dz89 DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4aeae184-01f9-43a5-864d-e5ad89841760/volumes/kubernetes.io~projected/kube-api-access-6wr2q DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~projected/kube-api-access-db858 DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/199ee6eb-216b-471a-8195-b9a71b842c16/volumes/kubernetes.io~projected/kube-api-access-9vlw9 DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:294 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-383 DeviceMajor:0 DeviceMinor:383 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/977c05aa-1b1d-4280-bd13-a098a44bb0a7/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/3f320485-3c0f-4555-947e-59bc38d66e7b/volumes/kubernetes.io~projected/kube-api-access-5dnsf DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/aece1cabd63e80915134bc17c73896485c34b6804c99c8bd61cecf65c8892f2d/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6520a4c2c2262de7fac1d87696f1c89946a5b3dd3b055d6d41b51fbcd443bd11/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/8320d815-b161-471e-97d6-dda571fef137/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0db914346bbfd7036fbd993406ba9a5b13c43d1e31c3024545aed41626bbf61b/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09fc78aeeec56c21a341c6f57f1f5ea83acdc76d91b55262c5ef553f92d2529f/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/268fe774-ac5a-405e-8dc9-4c2f3ec9c993/volumes/kubernetes.io~projected/kube-api-access-qqgxp DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:261 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/7b75558c-ebeb-4ae6-a623-676f3daef4f5/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82c99f3e171bd27b8819b6fb220469ee14c2d624b5fd9ecc937285d191647b15/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c8fad824-c375-4d0d-86aa-c1742b281b92/volumes/kubernetes.io~projected/kube-api-access-hd7xg DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/82812395-be27-46dd-8b16-bfab2c7fc22e/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-244 DeviceMajor:0 DeviceMinor:244 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-314 DeviceMajor:0 DeviceMinor:314 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-324 DeviceMajor:0 DeviceMinor:324 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0db914346bbfd70 MacAddress:d2:da:fd:a7:f7:83 Speed:10000 Mtu:8901} {Name:1b82444d9baec3b MacAddress:5a:55:20:93:ca:de Speed:10000 Mtu:8901} {Name:3d01ede25f56936 MacAddress:26:fd:1c:42:e5:b5 Speed:10000 Mtu:8901} {Name:82c99f3e171bd27 MacAddress:3e:92:d6:74:9a:6f Speed:10000 Mtu:8901} {Name:8e956ad15a1003f MacAddress:2e:53:73:10:ed:49 Speed:10000 Mtu:8901} {Name:8efc684afcc702a MacAddress:e6:e1:ca:a6:46:aa Speed:10000 Mtu:8901} {Name:98f4b4dcfb7c968 MacAddress:82:14:c6:a4:14:45 Speed:10000 Mtu:8901} {Name:a9c44857ca94da6 MacAddress:a2:ef:bb:67:4c:0b Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:da:53:a9:8d Speed:0 Mtu:9001} {Name:br-int MacAddress:6a:31:4f:c2:79:48 Speed:0 Mtu:8901} {Name:e05aea47c7faba1 MacAddress:22:f7:2c:cf:b1:f9 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:da:53:a9:8d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:2a:fe:68:f2:ca Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:6a:bf:b9:66:dd Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.618643 6171 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.618705 6171 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619028 6171 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619072 6171 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-35.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619253 6171 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619266 6171 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619280 6171 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619305 6171 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619395 6171 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619620 6171 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619776 6171 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619820 6171 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619834 6171 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619847 6171 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619863 6171 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.619874 6171 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.620779 6171 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.620792 6171 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:32:21.623105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.621265 6171 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:32:21.624013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.623999 6171 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:32:21.624013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624013 6171 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:32:21.624129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624119 6171 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:32:21.624308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624298 6171 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:32:21.624341 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624327 6171 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:32:21.624446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624433 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:32:21.624446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624448 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624454 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624460 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624465 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624471 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624477 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624485 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624494 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624499 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624507 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624517 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624542 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:32:21.624548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.624551 6171 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:32:21.628213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.628199 6171 server.go:1263] "Started kubelet" Oct 30 05:32:21.628316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.628251 6171 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:32:21.628433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.628358 6171 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:32:21.630502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.628532 6171 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:32:21.630502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.630212 6171 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:32:21.628922 ip-10-0-136-35 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:32:21.634214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.634200 6171 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:32:21.634789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.634772 6171 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:32:21.635069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.634948 6171 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:32:21.635069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.634996 6171 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:26:13 +0000 UTC" deadline="2028-05-31 10:55:05.814634533 +0000 UTC" Oct 30 05:32:21.635069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.635024 6171 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13901h22m44.179613992s" Oct 30 05:32:21.635069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.635034 6171 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:32:21.635069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.635066 6171 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:32:21.635342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.635091 6171 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:32:21.635342 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.635149 6171 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-35.ec2.internal\" not found" Oct 30 05:32:21.637471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.636995 6171 factory.go:55] Registering systemd factory Oct 30 05:32:21.637471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637016 6171 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:32:21.637471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637388 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config" seLinuxMountContext="" Oct 30 05:32:21.637471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637440 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016f2d58-bf54-4047-a2e0-8f14ee94825c" volumeName="kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs" seLinuxMountContext="" Oct 30 05:32:21.637471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637455 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="199ee6eb-216b-471a-8195-b9a71b842c16" volumeName="kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637491 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f320485-3c0f-4555-947e-59bc38d66e7b" volumeName="kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637505 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637515 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637532 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637545 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637580 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20562d0-453c-476a-9556-d5ee0ec52041" volumeName="kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637620 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ecb3cba-e0f3-4342-bf46-da748d315f0b" volumeName="kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637636 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637649 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637671 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fad824-c375-4d0d-86aa-c1742b281b92" volumeName="kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637686 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c19260-4322-43b7-8529-f7c520ad4feb" volumeName="kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637700 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637714 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae0fc4-ffe9-45fa-9367-3cb6307496ae" volumeName="kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637726 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="19db8a82-fab8-4558-8fd0-727d660a23c5" volumeName="kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637737 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aeae184-01f9-43a5-864d-e5ad89841760" volumeName="kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q" seLinuxMountContext="" Oct 30 05:32:21.637759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637748 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ecb3cba-e0f3-4342-bf46-da748d315f0b" volumeName="kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637759 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fad824-c375-4d0d-86aa-c1742b281b92" volumeName="kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637784 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637798 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="268fe774-ac5a-405e-8dc9-4c2f3ec9c993" volumeName="kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637815 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5690cc6f-94f7-4bc0-b271-35c91580a7e8" volumeName="kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637827 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637840 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637851 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637862 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637875 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20562d0-453c-476a-9556-d5ee0ec52041" volumeName="kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637891 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e6c19260-4322-43b7-8529-f7c520ad4feb" volumeName="kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637902 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="199ee6eb-216b-471a-8195-b9a71b842c16" volumeName="kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637912 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637926 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637942 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151d83c7-e9e4-4e60-ad4b-591771415adc" volumeName="kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637953 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151d83c7-e9e4-4e60-ad4b-591771415adc" volumeName="kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637965 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151d83c7-e9e4-4e60-ad4b-591771415adc" volumeName="kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637976 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f320485-3c0f-4555-947e-59bc38d66e7b" volumeName="kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.637987 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638012 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638024 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638036 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638067 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aeae184-01f9-43a5-864d-e5ad89841760" volumeName="kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638078 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638103 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638115 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638119 6171 factory.go:153] Registering CRI-O factory Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638126 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638140 6171 factory.go:223] Registration of the crio container factory successfully Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638153 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638165 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20562d0-453c-476a-9556-d5ee0ec52041" volumeName="kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638178 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="016f2d58-bf54-4047-a2e0-8f14ee94825c" volumeName="kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638199 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f320485-3c0f-4555-947e-59bc38d66e7b" volumeName="kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638214 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ecb3cba-e0f3-4342-bf46-da748d315f0b" volumeName="kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638229 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8320d815-b161-471e-97d6-dda571fef137" volumeName="kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638245 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="05b1f0ad-3ffd-4397-931e-60f5e9c69cc8" volumeName="kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638259 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638273 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638287 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638298 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8320d815-b161-471e-97d6-dda571fef137" volumeName="kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638358 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638373 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5ecb3cba-e0f3-4342-bf46-da748d315f0b" volumeName="kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638385 6171 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638413 6171 factory.go:103] Registering Raw factory Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638429 6171 manager.go:1199] Started watching for new ooms in manager Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638386 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8ae0fc4-ffe9-45fa-9367-3cb6307496ae" volumeName="kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638468 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3f320485-3c0f-4555-947e-59bc38d66e7b" volumeName="kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638510 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:32:21.638492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638523 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20562d0-453c-476a-9556-d5ee0ec52041" volumeName="kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638541 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d20562d0-453c-476a-9556-d5ee0ec52041" volumeName="kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638549 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="151d83c7-e9e4-4e60-ad4b-591771415adc" volumeName="kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638557 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638566 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638573 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" volumeName="kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638580 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8fad824-c375-4d0d-86aa-c1742b281b92" volumeName="kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638588 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="268fe774-ac5a-405e-8dc9-4c2f3ec9c993" volumeName="kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638595 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4aeae184-01f9-43a5-864d-e5ad89841760" volumeName="kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638602 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7b75558c-ebeb-4ae6-a623-676f3daef4f5" volumeName="kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638613 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="82812395-be27-46dd-8b16-bfab2c7fc22e" volumeName="kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638625 6171 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="977c05aa-1b1d-4280-bd13-a098a44bb0a7" volumeName="kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638634 6171 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638638 6171 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:32:21.640938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.638845 6171 manager.go:319] Starting recovery of all containers Oct 30 05:32:21.661291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.661218 6171 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:32:21.663086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.663065 6171 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:32:21.663086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.663085 6171 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:32:21.663237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.663162 6171 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:32:21.663237 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.663211 6171 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683339 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="8a93516ac4fe4e0eaf09f81704dcd36604f330a953640bf5c675a6934d2bc590" exitCode=0 Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683363 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="10118ab99bc6ce556b1ec52b99b23769ce2f5d1f2fc1ea3829f74d18b215153a" exitCode=0 Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683369 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="1091e33ce1eef1bafb52400ad757f996ac7a50566abbb3216cda689eb5371f73" exitCode=0 Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683375 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="700d346a219e7a9c0729b7986922cf86c6423371ac7dd4b18e5c97a8239b0422" exitCode=0 Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683380 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="794a7ef99712d2746358009372e7df4146f3ec2f4d2b5fcc02994e12477caed8" exitCode=0 Oct 30 05:32:21.683367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.683386 6171 generic.go:356] "Generic (PLEG): container finished" podID="3f320485-3c0f-4555-947e-59bc38d66e7b" containerID="d3a992a0354cd6f10f7134df86102cf66bb2abc9b8f667ded3d3cebd4c423311" exitCode=0 Oct 30 05:32:21.684827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.684800 6171 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:21.695215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.695193 6171 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:21.695315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.695226 6171 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:32:21.700454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.700415 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb" exitCode=0 Oct 30 05:32:21.702742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.702718 6171 generic.go:356] "Generic (PLEG): container finished" podID="82812395-be27-46dd-8b16-bfab2c7fc22e" containerID="d90d4a31ebac33ced284ff0808cc74304a1782caeb8e431a5ed4fa6b05b0558b" exitCode=0 Oct 30 05:32:21.707633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.707615 6171 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:21.707764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.707741 6171 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:32:21.717563 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.717543 6171 generic.go:356] "Generic (PLEG): container finished" podID="51317ae8aea2bacdb65b75f627da0c48" containerID="e2f1df8d09984d82ae14261ebf67210b51e1dd4a157eeeda5fbd1d8451fe8fcf" exitCode=0 Oct 30 05:32:21.739118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.739102 6171 manager.go:324] Recovery completed Oct 30 05:32:21.759787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759732 6171 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:32:21.759787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759749 6171 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:32:21.759787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759768 6171 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:32:21.759932 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759915 6171 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:32:21.759962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759931 6171 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:32:21.759962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759953 6171 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:32:21.759962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759961 6171 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:32:21.760037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759969 6171 policy_none.go:50] "Start" Oct 30 05:32:21.760037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759981 6171 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:32:21.760037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.759994 6171 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:32:21.760163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.760151 6171 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:32:21.760192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.760163 6171 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:32:21.760192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.760169 6171 policy_none.go:44] "Start" Oct 30 05:32:21.763790 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.763770 6171 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:32:21.764443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.764412 6171 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:32:21.767438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.767415 6171 manager.go:341] "Starting Device Plugin manager" Oct 30 05:32:21.768542 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.767440 6171 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:32:21.768586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.768546 6171 server.go:87] "Starting device plugin registration server" Oct 30 05:32:21.768812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.768801 6171 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:32:21.768843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.768817 6171 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:32:21.768911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.768896 6171 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:32:21.769010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.768979 6171 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:32:21.769117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.769018 6171 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:32:21.769117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.769025 6171 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:32:21.769211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.769093 6171 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:32:21.769004118Z","UUID":"6e96f0ce-0ed0-4299-aec2-41624f06c177","Handler":null,"Name":"","Endpoint":""} Oct 30 05:32:21.770492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.770474 6171 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:32:21.770572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.770499 6171 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:32:21.770572 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.770562 6171 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:32:21.869345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.869309 6171 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:32:21.871332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.871318 6171 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:32:21.871397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.871343 6171 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:32:21.871397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.871353 6171 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-35.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:32:21.871397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.871389 6171 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.907501 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.907480 6171 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.907583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.907537 6171 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.964394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.964345 6171 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal"] Oct 30 05:32:21.965032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965001 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"75abb82a1e3a44a283c8687c87965bd9","Type":"ContainerStarted","Data":"d80f23f796fd1d85d5a0676dfc4b882f5fa914fac5d094baf50e74494d67e75e"} Oct 30 05:32:21.965098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965059 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" event={"ID":"75abb82a1e3a44a283c8687c87965bd9","Type":"ContainerStarted","Data":"318caf3e39ba79dfb75d8e53b36db12ff051f56ff8f17c31599a9b9363f9d794"} Oct 30 05:32:21.965098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965076 6171 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.965162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965076 6171 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.965240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerStarted","Data":"b1dc29fbd0b429ebc3938bf20aaef9d9846a01d6bf06d6509522c83ecdaa902b"} Oct 30 05:32:21.965284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965267 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerDied","Data":"e2f1df8d09984d82ae14261ebf67210b51e1dd4a157eeeda5fbd1d8451fe8fcf"} Oct 30 05:32:21.965324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.965289 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" event={"ID":"51317ae8aea2bacdb65b75f627da0c48","Type":"ContainerStarted","Data":"5cd2ce73a6892d66df9bc4cf9506ccac67aa32398109b28b7de8280b5f0fe79e"} Oct 30 05:32:21.990661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.990642 6171 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:21.990723 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.990682 6171 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:21.991292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:21.991278 6171 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:21.991335 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:21.991317 6171 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.096744 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.096693 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.096744 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.096724 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.096744 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.096741 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197344 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197398 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197439 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197503 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/75abb82a1e3a44a283c8687c87965bd9-config\") pod \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" (UID: \"75abb82a1e3a44a283c8687c87965bd9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.197522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.197517 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/51317ae8aea2bacdb65b75f627da0c48-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" (UID: \"51317ae8aea2bacdb65b75f627da0c48\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.620931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.620902 6171 apiserver.go:52] "Watching apiserver" Oct 30 05:32:22.660287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.660264 6171 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:32:22.660943 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.660921 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-hh7pw","openshift-dns/dns-default-5pf44","openshift-ingress-canary/ingress-canary-4xwsj","openshift-multus/network-metrics-daemon-6bv7l","openshift-ovn-kubernetes/ovnkube-node-n79z7","openshift-monitoring/thanos-querier-7d56d56779-5qhql","kube-system/konnectivity-agent-gnmzd","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal","openshift-multus/multus-wl429","kube-system/global-pull-secret-syncer-f8w5w","openshift-insights/insights-runtime-extractor-4tpzp","openshift-dns/node-resolver-bv9np","kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal","openshift-monitoring/metrics-server-689db8797f-sb5b8","openshift-cluster-node-tuning-operator/tuned-xprpl","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq","openshift-monitoring/alertmanager-main-0","openshift-network-diagnostics/network-check-target-kvzzn","openshift-monitoring/node-exporter-nlj4r","openshift-multus/multus-additional-cni-plugins-98xfs","openshift-network-operator/iptables-alerter-ttvn4"] Oct 30 05:32:22.678379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678357 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.678534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678420 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:32:22.678534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678432 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:32:22.678534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678425 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.678534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678486 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:32:22.678776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678728 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-2ztbt\"" Oct 30 05:32:22.678776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678772 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:32:22.678871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.678826 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-f8j26\"" Oct 30 05:32:22.684992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.684974 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:32:22.685597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.685583 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.685683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.685662 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.685752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.685740 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.686461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.686447 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.686585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.686572 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:32:22.686771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.686756 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.687038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.687024 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:32:22.687129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.687064 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:32:22.687129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.687076 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.687129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.687113 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.691139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.691125 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.691213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.691202 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:32:22.691577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.691564 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.691613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.691565 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:32:22.691613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.691590 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.692998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.692976 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.692998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.692992 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.693145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693057 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.693190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693153 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.693253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693231 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.693321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693269 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.693368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693256 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:32:22.693368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693356 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.693466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693430 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:32:22.693466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693454 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:32:22.693746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693732 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:32:22.693801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693749 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.693892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693875 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:32:22.693944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693884 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:32:22.693991 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.693948 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:32:22.694169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694143 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:32:22.694293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694198 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zsr6d\"" Oct 30 05:32:22.694293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694200 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:32:22.694293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694215 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:32:22.694293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694226 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:32:22.694448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694319 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:32:22.695005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.694988 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:32:22.695233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.695215 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:32:22.695314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.695219 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-jtd4z\"" Oct 30 05:32:22.696734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.696716 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:32:22.698609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.698590 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-nr89r\"" Oct 30 05:32:22.698882 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.698868 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:32:22.698947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.698937 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-k22sc\"" Oct 30 05:32:22.699008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.698962 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-xpx6p\"" Oct 30 05:32:22.699008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699001 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699018 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699022 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-jvn8d\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699037 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699019 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-7pqs8\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699083 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:32:22.699126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.699104 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:32:22.702351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702324 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:32:22.702351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702347 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702386 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702398 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-kbh7d\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702404 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702390 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-e6iddlm2f0hh2\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702421 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702445 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702433 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-v8jj4\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702459 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702446 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-bs12it7als558\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702420 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:32:22.702482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702444 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:32:22.702793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702491 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-55j5k\"" Oct 30 05:32:22.702793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.702526 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7pqn7\"" Oct 30 05:32:22.703260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703248 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-wks87\"" Oct 30 05:32:22.703299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703292 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-vg6rp\"" Oct 30 05:32:22.703299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703292 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-gf7fl\"" Oct 30 05:32:22.703374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703303 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mjhr7\"" Oct 30 05:32:22.703485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703469 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:32:22.703541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703475 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:32:22.703541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703494 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lg7qg\"" Oct 30 05:32:22.703541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.703496 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bmrwn\"" Oct 30 05:32:22.705124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.705111 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:32:22.706343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.706329 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:32:22.722559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.722545 6171 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.722703 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.722691 6171 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.736787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.736771 6171 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:32:22.745263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.745246 6171 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:22.745319 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:22.745284 6171 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-35.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.745920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.745907 6171 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:32:22.745967 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:22.745937 6171 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-35.ec2.internal" Oct 30 05:32:22.799291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799274 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.799386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799298 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqgxp\" (UniqueName: \"kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.799386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799316 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.799386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799333 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.799386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799350 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.799386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799368 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799406 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799434 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799453 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799458 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-textfile\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799493 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799536 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799579 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-client-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799617 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.799647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799646 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/977c05aa-1b1d-4280-bd13-a098a44bb0a7-audit-log\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-metrics-server-audit-profiles\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799710 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799737 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b75558c-ebeb-4ae6-a623-676f3daef4f5-tmp\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799758 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799778 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-env-overrides\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799788 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799811 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799843 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799867 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hswx9\" (UniqueName: \"kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799895 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799915 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.799987 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800083 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-db858\" (UniqueName: \"kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800091 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800103 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-tls\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800113 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800130 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800152 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800168 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800183 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800202 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dnsf\" (UniqueName: \"kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800228 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dz89\" (UniqueName: \"kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800252 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800271 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-grpc-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800285 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800309 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800329 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/016f2d58-bf54-4047-a2e0-8f14ee94825c-metrics-certs\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800333 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800351 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/151d83c7-e9e4-4e60-ad4b-591771415adc-config-volume\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800394 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800456 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8320d815-b161-471e-97d6-dda571fef137-agent-certs\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800490 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800516 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800522 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-tls\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800532 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800546 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800606 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800626 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800637 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800664 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800682 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800712 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800726 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.800721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800742 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800764 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800814 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800834 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-etc-tuned\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800837 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800860 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4aeae184-01f9-43a5-864d-e5ad89841760-tmp\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800877 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800909 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800916 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/977c05aa-1b1d-4280-bd13-a098a44bb0a7-secret-metrics-server-client-certs\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800949 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.800948 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d20562d0-453c-476a-9556-d5ee0ec52041-ovn-node-metrics-cert\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801039 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801079 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801186 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801208 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801212 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801224 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801241 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801260 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801278 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wr2q\" (UniqueName: \"kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801292 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801305 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801319 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801340 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801360 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-cni-binary-copy\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801363 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801406 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-daemon-config\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801441 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801470 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801488 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-iptables-alerter-script\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801504 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801521 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801539 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/977c05aa-1b1d-4280-bd13-a098a44bb0a7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801561 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/151d83c7-e9e4-4e60-ad4b-591771415adc-metrics-tls\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801561 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-cert\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801607 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801642 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801661 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801676 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801700 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801741 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801771 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801795 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801822 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801846 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801873 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801911 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801953 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801990 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.801959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.801995 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5ecb3cba-e0f3-4342-bf46-da748d315f0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802007 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802103 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802144 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/7b75558c-ebeb-4ae6-a623-676f3daef4f5-secret-thanos-querier-tls\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802194 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802223 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802250 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802300 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-accelerators-collector-config\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802347 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802375 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87dv7\" (UniqueName: \"kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802404 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802415 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b75558c-ebeb-4ae6-a623-676f3daef4f5-metrics-client-ca\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802414 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5690cc6f-94f7-4bc0-b271-35c91580a7e8-original-pull-secret\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802429 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vlw9\" (UniqueName: \"kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802527 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802552 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802588 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802622 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802648 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802663 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802709 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd7xg\" (UniqueName: \"kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802773 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802821 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802851 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802878 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802991 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-config\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.802990 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803087 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803112 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803128 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5ecb3cba-e0f3-4342-bf46-da748d315f0b-data-volume\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803165 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803188 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803212 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803132 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803247 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803278 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/3f320485-3c0f-4555-947e-59bc38d66e7b-cni-binary-copy\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803284 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803284 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803297 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxwr4\" (UniqueName: \"kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803339 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803301 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c19260-4322-43b7-8529-f7c520ad4feb-tmp-dir\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803359 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803406 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbj44\" (UniqueName: \"kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803421 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803453 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803478 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803504 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/199ee6eb-216b-471a-8195-b9a71b842c16-serviceca\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803507 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803559 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803599 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/82812395-be27-46dd-8b16-bfab2c7fc22e-metrics-client-ca\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803612 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/151d83c7-e9e4-4e60-ad4b-591771415adc-tmp-dir\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803628 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803646 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803660 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803740 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803748 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vkmr\" (UniqueName: \"kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803749 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8320d815-b161-471e-97d6-dda571fef137-konnectivity-ca\") pod \"konnectivity-agent-gnmzd\" (UID: \"8320d815-b161-471e-97d6-dda571fef137\") " pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803781 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803811 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjtqk\" (UniqueName: \"kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803829 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803887 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803966 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d20562d0-453c-476a-9556-d5ee0ec52041-ovnkube-script-lib\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.803968 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.804004 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.804085 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc8b7\" (UniqueName: \"kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.804152 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxr4c\" (UniqueName: \"kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.804210 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.804252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.804232 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.821132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.821107 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hswx9\" (UniqueName: \"kubernetes.io/projected/82812395-be27-46dd-8b16-bfab2c7fc22e-kube-api-access-hswx9\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.821203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.821131 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqgxp\" (UniqueName: \"kubernetes.io/projected/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-kube-api-access-qqgxp\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.821203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.821192 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dnsf\" (UniqueName: \"kubernetes.io/projected/3f320485-3c0f-4555-947e-59bc38d66e7b-kube-api-access-5dnsf\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.826799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.826784 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wr2q\" (UniqueName: \"kubernetes.io/projected/4aeae184-01f9-43a5-864d-e5ad89841760-kube-api-access-6wr2q\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.827005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.826985 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dz89\" (UniqueName: \"kubernetes.io/projected/d20562d0-453c-476a-9556-d5ee0ec52041-kube-api-access-5dz89\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.827189 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.827175 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"alertmanager-main-0\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:22.827562 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.827548 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87dv7\" (UniqueName: \"kubernetes.io/projected/151d83c7-e9e4-4e60-ad4b-591771415adc-kube-api-access-87dv7\") pod \"dns-default-5pf44\" (UID: \"151d83c7-e9e4-4e60-ad4b-591771415adc\") " pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:22.827613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.827598 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxwr4\" (UniqueName: \"kubernetes.io/projected/19db8a82-fab8-4558-8fd0-727d660a23c5-kube-api-access-sxwr4\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.828072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.828057 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd7xg\" (UniqueName: \"kubernetes.io/projected/c8fad824-c375-4d0d-86aa-c1742b281b92-kube-api-access-hd7xg\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.831674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.831658 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vlw9\" (UniqueName: \"kubernetes.io/projected/199ee6eb-216b-471a-8195-b9a71b842c16-kube-api-access-9vlw9\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.832812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.832796 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vkmr\" (UniqueName: \"kubernetes.io/projected/5ecb3cba-e0f3-4342-bf46-da748d315f0b-kube-api-access-8vkmr\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.832966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.832950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-db858\" (UniqueName: \"kubernetes.io/projected/7b75558c-ebeb-4ae6-a623-676f3daef4f5-kube-api-access-db858\") pod \"thanos-querier-7d56d56779-5qhql\" (UID: \"7b75558c-ebeb-4ae6-a623-676f3daef4f5\") " pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:22.833659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.833629 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbj44\" (UniqueName: \"kubernetes.io/projected/016f2d58-bf54-4047-a2e0-8f14ee94825c-kube-api-access-bbj44\") pod \"network-metrics-daemon-6bv7l\" (UID: \"016f2d58-bf54-4047-a2e0-8f14ee94825c\") " pod="openshift-multus/network-metrics-daemon-6bv7l" Oct 30 05:32:22.833659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.833646 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxr4c\" (UniqueName: \"kubernetes.io/projected/977c05aa-1b1d-4280-bd13-a098a44bb0a7-kube-api-access-mxr4c\") pod \"metrics-server-689db8797f-sb5b8\" (UID: \"977c05aa-1b1d-4280-bd13-a098a44bb0a7\") " pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:22.833802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.833629 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjtqk\" (UniqueName: \"kubernetes.io/projected/e6c19260-4322-43b7-8529-f7c520ad4feb-kube-api-access-fjtqk\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.833802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.833718 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmr6f\" (UniqueName: \"kubernetes.io/projected/05b1f0ad-3ffd-4397-931e-60f5e9c69cc8-kube-api-access-mmr6f\") pod \"network-check-target-kvzzn\" (UID: \"05b1f0ad-3ffd-4397-931e-60f5e9c69cc8\") " pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:22.834334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.834317 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc8b7\" (UniqueName: \"kubernetes.io/projected/c8ae0fc4-ffe9-45fa-9367-3cb6307496ae-kube-api-access-mc8b7\") pod \"ingress-canary-4xwsj\" (UID: \"c8ae0fc4-ffe9-45fa-9367-3cb6307496ae\") " pod="openshift-ingress-canary/ingress-canary-4xwsj" Oct 30 05:32:22.904914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904851 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.904914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904890 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904915 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904940 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904962 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904971 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-system-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904984 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904971 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905004 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-netns\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905003 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-systemd-units\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.904975 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-kubernetes\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905007 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905068 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905067 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905026 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-sys\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-multus\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905114 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905132 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905151 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905170 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-kubelet\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905178 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905207 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-cnibin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905209 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905235 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905259 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905263 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-var-lib-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905287 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905306 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905323 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5ecb3cba-e0f3-4342-bf46-da748d315f0b-crio-socket\") pod \"insights-runtime-extractor-4tpzp\" (UID: \"5ecb3cba-e0f3-4342-bf46-da748d315f0b\") " pod="openshift-insights/insights-runtime-extractor-4tpzp" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905335 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905315 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-cni-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905351 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905370 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/199ee6eb-216b-471a-8195-b9a71b842c16-host\") pod \"node-ca-hh7pw\" (UID: \"199ee6eb-216b-471a-8195-b9a71b842c16\") " pod="openshift-image-registry/node-ca-hh7pw" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905375 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905206 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-var-lib-kubelet\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905383 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-lib-modules\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905391 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-systemd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905394 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-node-exporter-wtmp\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905400 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905412 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-multus-certs\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905425 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-device-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905184 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-system-cni-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905428 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-socket-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905439 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905462 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905477 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905493 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905496 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-etc-kubernetes\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905513 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905525 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905508 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-host\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905533 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905550 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-hostroot\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905534 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905561 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905563 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905600 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905625 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905652 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905657 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-dbus\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.905635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905690 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/3f320485-3c0f-4555-947e-59bc38d66e7b-cnibin\") pod \"multus-additional-cni-plugins-98xfs\" (UID: \"3f320485-3c0f-4555-947e-59bc38d66e7b\") " pod="openshift-multus/multus-additional-cni-plugins-98xfs" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905720 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-systemd\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905731 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905734 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysconfig\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905754 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905769 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905774 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-socket-dir-parent\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905785 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905801 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905813 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-node-log\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905818 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-run-ovn\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905815 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905834 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-bin\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905843 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-slash\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905843 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905867 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905868 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-var-lib-cni-bin\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905866 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-registration-dir\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905883 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905900 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-etc-openvswitch\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905905 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905907 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5690cc6f-94f7-4bc0-b271-35c91580a7e8-kubelet-config\") pod \"global-pull-secret-syncer-f8w5w\" (UID: \"5690cc6f-94f7-4bc0-b271-35c91580a7e8\") " pod="kube-system/global-pull-secret-syncer-f8w5w" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905922 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-cni-netd\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/19db8a82-fab8-4558-8fd0-727d660a23c5-sys-fs\") pod \"aws-ebs-csi-driver-node-68wkq\" (UID: \"19db8a82-fab8-4558-8fd0-727d660a23c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-68wkq" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905954 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905978 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905978 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-sys\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905997 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-host-run-k8s-cni-cncf-io\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.905993 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906015 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-kubelet\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906029 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906073 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906088 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-log-socket\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906097 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906120 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906127 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/82812395-be27-46dd-8b16-bfab2c7fc22e-root\") pod \"node-exporter-nlj4r\" (UID: \"82812395-be27-46dd-8b16-bfab2c7fc22e\") " pod="openshift-monitoring/node-exporter-nlj4r" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906124 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-run\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906149 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c8fad824-c375-4d0d-86aa-c1742b281b92-multus-conf-dir\") pod \"multus-wl429\" (UID: \"c8fad824-c375-4d0d-86aa-c1742b281b92\") " pod="openshift-multus/multus-wl429" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906166 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906181 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906192 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d20562d0-453c-476a-9556-d5ee0ec52041-host-run-netns\") pod \"ovnkube-node-n79z7\" (UID: \"d20562d0-453c-476a-9556-d5ee0ec52041\") " pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906195 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906214 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906220 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/268fe774-ac5a-405e-8dc9-4c2f3ec9c993-host-slash\") pod \"iptables-alerter-ttvn4\" (UID: \"268fe774-ac5a-405e-8dc9-4c2f3ec9c993\") " pod="openshift-network-operator/iptables-alerter-ttvn4" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906231 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906279 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6c19260-4322-43b7-8529-f7c520ad4feb-hosts-file\") pod \"node-resolver-bv9np\" (UID: \"e6c19260-4322-43b7-8529-f7c520ad4feb\") " pod="openshift-dns/node-resolver-bv9np" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-sysctl-conf\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:22.907441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:22.906295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4aeae184-01f9-43a5-864d-e5ad89841760-etc-modprobe-d\") pod \"tuned-xprpl\" (UID: \"4aeae184-01f9-43a5-864d-e5ad89841760\") " pod="openshift-cluster-node-tuning-operator/tuned-xprpl" Oct 30 05:32:23.628464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:23.628437 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7d56d56779-5qhql" Oct 30 05:32:23.818725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:23.818693 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5pf44" Oct 30 05:32:25.069883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:25.069849 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:26.379088 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:26.379058 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:32:26.502256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:26.502229 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:30.285221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:30.285184 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:30.285697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:30.285399 6171 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:32:40.998581 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:40.998506 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-689db8797f-sb5b8" Oct 30 05:32:43.202386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:43.202353 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gnmzd" Oct 30 05:32:47.036442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036402 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:47.036878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036785 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="alertmanager" containerID="cri-o://5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78" gracePeriod=120 Oct 30 05:32:47.036878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036812 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-metric" containerID="cri-o://06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24" gracePeriod=120 Oct 30 05:32:47.037004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036855 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="config-reloader" containerID="cri-o://f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79" gracePeriod=120 Oct 30 05:32:47.037004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036876 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="prom-label-proxy" containerID="cri-o://d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651" gracePeriod=120 Oct 30 05:32:47.037004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036833 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-web" containerID="cri-o://3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89" gracePeriod=120 Oct 30 05:32:47.037187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.036889 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy" containerID="cri-o://5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c" gracePeriod=120 Oct 30 05:32:47.242233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.242210 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-n79z7" Oct 30 05:32:47.800082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800029 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651" exitCode=0 Oct 30 05:32:47.800082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800072 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24" exitCode=0 Oct 30 05:32:47.800082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800081 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c" exitCode=0 Oct 30 05:32:47.800082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800087 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79" exitCode=0 Oct 30 05:32:47.800082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800076 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651"} Oct 30 05:32:47.800436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800092 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78" exitCode=0 Oct 30 05:32:47.800436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800172 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24"} Oct 30 05:32:47.800436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800211 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c"} Oct 30 05:32:47.800436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800248 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79"} Oct 30 05:32:47.800436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:47.800284 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78"} Oct 30 05:32:48.267609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.267585 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.466749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.466668 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.466749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.466721 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.466749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.466752 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.466962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.466921 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.466977 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467075 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467112 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467146 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467172 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:48.467280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467183 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467215 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467267 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467310 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467345 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") pod \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\" (UID: \"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f\") " Oct 30 05:32:48.467569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467553 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-metrics-client-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.468271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.467984 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:48.468271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.468182 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:32:48.469957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.469929 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.470394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.470341 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.471096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471065 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.471195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:32:48.471195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.471426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471409 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:48.471504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471483 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.471557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.471532 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "kube-api-access-56cpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:32:48.474901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.474878 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.481152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.481131 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config" pod "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" (UID: "b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:32:48.568465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568425 6171 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-main-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568465 6171 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568475 6171 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568487 6171 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568496 6171 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-out\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568505 6171 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-cluster-tls-config\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568513 6171 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568522 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56cpm\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-kube-api-access-56cpm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568530 6171 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568539 6171 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-tls-assets\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568546 6171 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-web-config\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.568593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.568553 6171 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f-alertmanager-main-db\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:32:48.806666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.806640 6171 generic.go:356] "Generic (PLEG): container finished" podID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerID="3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89" exitCode=0 Oct 30 05:32:48.806799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.806722 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89"} Oct 30 05:32:48.806799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.806787 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f","Type":"ContainerDied","Data":"3d01ede25f5693614f6cd46c4bd1b2d40c5ad56dc2485234fe47579c77a2a167"} Oct 30 05:32:48.806908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.806812 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.806908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.806823 6171 scope.go:122] "RemoveContainer" containerID="d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651" Oct 30 05:32:48.814454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.814436 6171 scope.go:122] "RemoveContainer" containerID="06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24" Oct 30 05:32:48.820860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.820842 6171 scope.go:122] "RemoveContainer" containerID="5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c" Oct 30 05:32:48.827213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.827196 6171 scope.go:122] "RemoveContainer" containerID="3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89" Oct 30 05:32:48.833565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.833548 6171 scope.go:122] "RemoveContainer" containerID="f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79" Oct 30 05:32:48.840168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.840154 6171 scope.go:122] "RemoveContainer" containerID="5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78" Oct 30 05:32:48.846515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.846499 6171 scope.go:122] "RemoveContainer" containerID="47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb" Oct 30 05:32:48.852880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.852862 6171 scope.go:122] "RemoveContainer" containerID="d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651" Oct 30 05:32:48.853149 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.853131 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651\": container with ID starting with d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651 not found: ID does not exist" containerID="d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651" Oct 30 05:32:48.853203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853157 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651"} err="failed to get container status \"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651\": rpc error: code = NotFound desc = could not find container \"d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651\": container with ID starting with d37d5e0798509953c6c841212773dd3a21d7594f420e13d78afae0b1f1505651 not found: ID does not exist" Oct 30 05:32:48.853203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853191 6171 scope.go:122] "RemoveContainer" containerID="06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24" Oct 30 05:32:48.853451 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.853433 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24\": container with ID starting with 06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24 not found: ID does not exist" containerID="06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24" Oct 30 05:32:48.853487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853459 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24"} err="failed to get container status \"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24\": rpc error: code = NotFound desc = could not find container \"06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24\": container with ID starting with 06a089598a11ac46c96e7b81bc24b100d34465aefef8f5889757064249d80c24 not found: ID does not exist" Oct 30 05:32:48.853487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853475 6171 scope.go:122] "RemoveContainer" containerID="5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c" Oct 30 05:32:48.853686 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.853668 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c\": container with ID starting with 5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c not found: ID does not exist" containerID="5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c" Oct 30 05:32:48.853727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853692 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c"} err="failed to get container status \"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c\": rpc error: code = NotFound desc = could not find container \"5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c\": container with ID starting with 5efe2aaa13725ec2a5369aa6e287ee06284e03179428545a792078b86a78783c not found: ID does not exist" Oct 30 05:32:48.853727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853707 6171 scope.go:122] "RemoveContainer" containerID="3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89" Oct 30 05:32:48.853894 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.853876 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89\": container with ID starting with 3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89 not found: ID does not exist" containerID="3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89" Oct 30 05:32:48.853938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853899 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89"} err="failed to get container status \"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89\": rpc error: code = NotFound desc = could not find container \"3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89\": container with ID starting with 3b46f798d3f4ec9bafa8d74b50a1e8e5bf0e4b5bcc6b6d0a9d6f9d78e5387e89 not found: ID does not exist" Oct 30 05:32:48.853938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.853910 6171 scope.go:122] "RemoveContainer" containerID="f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79" Oct 30 05:32:48.854159 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.854138 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79\": container with ID starting with f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79 not found: ID does not exist" containerID="f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79" Oct 30 05:32:48.854229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.854162 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79"} err="failed to get container status \"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79\": rpc error: code = NotFound desc = could not find container \"f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79\": container with ID starting with f478aae538e691f91071ff06235685e21ff6c6bd6dce80d03b54357cc845eb79 not found: ID does not exist" Oct 30 05:32:48.854229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.854175 6171 scope.go:122] "RemoveContainer" containerID="5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78" Oct 30 05:32:48.854379 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.854363 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78\": container with ID starting with 5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78 not found: ID does not exist" containerID="5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78" Oct 30 05:32:48.854420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.854383 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78"} err="failed to get container status \"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78\": rpc error: code = NotFound desc = could not find container \"5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78\": container with ID starting with 5959c4d051bccf89805601cd5be2a0cceb462e3f60265b17fee17e1879b99a78 not found: ID does not exist" Oct 30 05:32:48.854420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.854394 6171 scope.go:122] "RemoveContainer" containerID="47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb" Oct 30 05:32:48.854630 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:48.854613 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb\": container with ID starting with 47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb not found: ID does not exist" containerID="47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb" Oct 30 05:32:48.854675 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.854635 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb"} err="failed to get container status \"47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb\": rpc error: code = NotFound desc = could not find container \"47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb\": container with ID starting with 47c1a3935a8b41b0e74c2aa3191ad8186ef627802435f8b54e0ca5e68355abcb not found: ID does not exist" Oct 30 05:32:48.870531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.870509 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:48.882613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.882592 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:48.908187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908163 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:48.908405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908391 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="prom-label-proxy" Oct 30 05:32:48.908405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908405 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="prom-label-proxy" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908420 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908425 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908434 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="alertmanager" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908445 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="alertmanager" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908454 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="init-config-reloader" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908458 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="init-config-reloader" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908470 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-web" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908474 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-web" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908483 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-metric" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908487 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-metric" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908492 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="config-reloader" Oct 30 05:32:48.908523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908496 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="config-reloader" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908535 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-web" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908542 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy-metric" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908547 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="config-reloader" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908553 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="prom-label-proxy" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908558 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="kube-rbac-proxy" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908562 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="alertmanager" Oct 30 05:32:48.908859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.908567 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" containerName="init-config-reloader" Oct 30 05:32:48.911150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.911136 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.929252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.929223 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:32:48.929252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.929233 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:32:48.929252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.929223 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:32:48.941263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941200 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:32:48.941263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941205 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:32:48.941263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941223 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:32:48.941486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941358 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-zsr6d\"" Oct 30 05:32:48.941486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941301 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:32:48.941486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941397 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:32:48.941592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.941565 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:32:48.942184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.942165 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:48.970956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.970932 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.970961 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58f9l\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-kube-api-access-58f9l\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.970983 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971002 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-web-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971073 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971128 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971155 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-config-out\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971226 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971243 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971318 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971353 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971373 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:48.971442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:48.971398 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071623 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071662 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-web-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071688 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071726 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071749 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-config-out\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071774 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.071828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071800 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071836 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071880 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071910 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071936 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071968 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.071996 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58f9l\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-kube-api-access-58f9l\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.072032 6171 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:32:49.072430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.072241 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.072901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.072814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.073319 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.073292 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2e82198e-beb3-4f85-965d-09b144b8d640-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075026 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075133 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075150 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-web-config\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075456 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075602 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075646 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-tls-assets\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.075733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.075687 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2e82198e-beb3-4f85-965d-09b144b8d640-config-out\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.076172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.076155 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.076771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.076754 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/2e82198e-beb3-4f85-965d-09b144b8d640-config-volume\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.094964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.094941 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58f9l\" (UniqueName: \"kubernetes.io/projected/2e82198e-beb3-4f85-965d-09b144b8d640-kube-api-access-58f9l\") pod \"alertmanager-main-0\" (UID: \"2e82198e-beb3-4f85-965d-09b144b8d640\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.226358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.226336 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:32:49.560727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.560698 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:32:49.578271 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:49.578245 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e82198e_beb3_4f85_965d_09b144b8d640.slice/crio-05172c3ab0f6f23140348e9756c6fd11d74b061ac66649c18d9262faee4c47ac WatchSource:0}: Error finding container 05172c3ab0f6f23140348e9756c6fd11d74b061ac66649c18d9262faee4c47ac: Status 404 returned error can't find the container with id 05172c3ab0f6f23140348e9756c6fd11d74b061ac66649c18d9262faee4c47ac Oct 30 05:32:49.673822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.673763 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f" path="/var/lib/kubelet/pods/b5bb2921-d4ab-417d-ad12-1a0c57e3ef4f/volumes" Oct 30 05:32:49.811184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.811117 6171 generic.go:356] "Generic (PLEG): container finished" podID="2e82198e-beb3-4f85-965d-09b144b8d640" containerID="18cc81e21f605c78f699877f25dfd1372440a6112bda3f37cb90d79b3eeb17ff" exitCode=0 Oct 30 05:32:49.811313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.811209 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerDied","Data":"18cc81e21f605c78f699877f25dfd1372440a6112bda3f37cb90d79b3eeb17ff"} Oct 30 05:32:49.811313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:49.811274 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"05172c3ab0f6f23140348e9756c6fd11d74b061ac66649c18d9262faee4c47ac"} Oct 30 05:32:50.818611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818574 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"09dc1c16951beb992ad994ab5448940edc8dd46f4554f5b1d3466b270a25a4a9"} Oct 30 05:32:50.819071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818664 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"e70fa5eb9141bd8fbcc100b40e0927e9f476f43b90992e00fbffb3981dc18cdb"} Oct 30 05:32:50.819071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818703 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"4b453ee95099e35a6798af1c1ada1982e3f9a7f39582a918afed0fa15a20731f"} Oct 30 05:32:50.819071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818729 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"f7299e8e192b1eaad0921f487007fc855d650f9d627cb788b811b890fb9bf2ee"} Oct 30 05:32:50.819071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818753 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"0e6ebc90d34741d9d5241515705ceb188dd11e26d39e87b256d14e29ca2c19e2"} Oct 30 05:32:50.819071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.818784 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"2e82198e-beb3-4f85-965d-09b144b8d640","Type":"ContainerStarted","Data":"f85c1014775b42e183afe4c807451bf9eb0732e5b469d67dd81157d9b02ff45f"} Oct 30 05:32:50.905507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.905396 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.893005226 podStartE2EDuration="2.905376876s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.37165ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:50.893218513 +0000 UTC m=+29.333595708" watchObservedRunningTime="2026-10-30 05:32:50.905376876 +0000 UTC m=+29.345754062" Oct 30 05:32:50.999014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:50.998984 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-665558d944-qjs57"] Oct 30 05:32:51.001075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.001061 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.015666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.015640 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 05:32:51.015666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.015658 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 05:32:51.015852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.015736 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 05:32:51.015852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.015775 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 05:32:51.015852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.015814 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qcfck\"" Oct 30 05:32:51.035054 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.035029 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-665558d944-qjs57"] Oct 30 05:32:51.085468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085414 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-serving-certs-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085450 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrjcv\" (UniqueName: \"kubernetes.io/projected/294487e3-1f88-4803-8b40-edc9c386ad50-kube-api-access-rrjcv\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085467 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085484 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085503 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085607 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-federate-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.085784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.085748 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-metrics-client-ca\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.105628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.105605 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 05:32:51.118204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.118178 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 05:32:51.186332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186309 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-federate-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186344 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186369 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-metrics-client-ca\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186402 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-serving-certs-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186427 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrjcv\" (UniqueName: \"kubernetes.io/projected/294487e3-1f88-4803-8b40-edc9c386ad50-kube-api-access-rrjcv\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186442 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186463 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.186505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.186489 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.187237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.187212 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-serving-certs-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.187422 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.187350 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-trusted-ca-bundle\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.187706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.187686 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/294487e3-1f88-4803-8b40-edc9c386ad50-metrics-client-ca\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.189211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.189191 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-federate-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.189466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.189445 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.189518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.189502 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.189557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.189505 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/294487e3-1f88-4803-8b40-edc9c386ad50-telemeter-client-tls\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.211116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.211096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrjcv\" (UniqueName: \"kubernetes.io/projected/294487e3-1f88-4803-8b40-edc9c386ad50-kube-api-access-rrjcv\") pod \"telemeter-client-665558d944-qjs57\" (UID: \"294487e3-1f88-4803-8b40-edc9c386ad50\") " pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.313389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.313368 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" Oct 30 05:32:51.647709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.647640 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-665558d944-qjs57"] Oct 30 05:32:51.651280 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:32:51.651252 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod294487e3_1f88_4803_8b40_edc9c386ad50.slice/crio-5fe768c382ca598213bf5915a3e43ef0242e51dc8e8e2a3ca9b81b798678dfff WatchSource:0}: Error finding container 5fe768c382ca598213bf5915a3e43ef0242e51dc8e8e2a3ca9b81b798678dfff: Status 404 returned error can't find the container with id 5fe768c382ca598213bf5915a3e43ef0242e51dc8e8e2a3ca9b81b798678dfff Oct 30 05:32:51.653179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.653162 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:32:51.825731 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:51.825692 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" event={"ID":"294487e3-1f88-4803-8b40-edc9c386ad50","Type":"ContainerStarted","Data":"5fe768c382ca598213bf5915a3e43ef0242e51dc8e8e2a3ca9b81b798678dfff"} Oct 30 05:32:53.834257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:53.834222 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" event={"ID":"294487e3-1f88-4803-8b40-edc9c386ad50","Type":"ContainerStarted","Data":"efd0dfed1226c73c6e2e5901c8a393121e5cb5f01868f5d3de955b9cfb34d4db"} Oct 30 05:32:53.834630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:53.834276 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" event={"ID":"294487e3-1f88-4803-8b40-edc9c386ad50","Type":"ContainerStarted","Data":"d175d56c4f3f7ac4498821409f4578cd5af990d745353c208d29ce36f6f6eee6"} Oct 30 05:32:53.834630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:53.834298 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" event={"ID":"294487e3-1f88-4803-8b40-edc9c386ad50","Type":"ContainerStarted","Data":"c9ff43ea963183108f98baf8959e4864510d238ad9f786eeb1434d1f884434bc"} Oct 30 05:32:53.905666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:53.905615 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-665558d944-qjs57" podStartSLOduration=2.340005481 podStartE2EDuration="3.905602379s" totalImagesPullingTime="1.565596898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:32:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:32:53.893675938 +0000 UTC m=+32.334053131" watchObservedRunningTime="2026-10-30 05:32:53.905602379 +0000 UTC m=+32.345979569" Oct 30 05:32:55.073215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:32:55.073188 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-kvzzn" Oct 30 05:32:56.379563 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:32:56.379525 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:33:26.378279 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:33:26.378243 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:33:56.378514 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:33:56.378476 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:34:26.379401 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:34:26.379366 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:34:56.377900 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:34:56.377856 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:35:26.378882 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:35:26.378843 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:35:56.377866 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:35:56.377817 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:36:15.203120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.203092 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl"] Oct 30 05:36:15.210810 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.210788 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.225289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.225270 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 05:36:15.237179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.237154 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-pc7fb\"" Oct 30 05:36:15.237179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.237173 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 05:36:15.238982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.238961 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl"] Oct 30 05:36:15.334376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.334351 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b48x6\" (UniqueName: \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.334511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.334388 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.334511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.334417 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.435608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.435585 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b48x6\" (UniqueName: \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.435687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.435621 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.435687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.435649 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.435955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.435939 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.436020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.435983 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.458998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.458950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b48x6\" (UniqueName: \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.525987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.525966 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:15.824390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:15.824357 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl"] Oct 30 05:36:15.827637 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:15.827612 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927095aa_0630_4030_ad8f_b21b1bd839b0.slice/crio-a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718 WatchSource:0}: Error finding container a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718: Status 404 returned error can't find the container with id a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718 Oct 30 05:36:16.493722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:16.493682 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" event={"ID":"927095aa-0630-4030-ad8f-b21b1bd839b0","Type":"ContainerStarted","Data":"a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718"} Oct 30 05:36:21.513840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:21.513805 6171 generic.go:356] "Generic (PLEG): container finished" podID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerID="c86a854b9bc62e33f778400d227b93a54efc3cbb47307a14958caa3c0cbede75" exitCode=0 Oct 30 05:36:21.514215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:21.513892 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" event={"ID":"927095aa-0630-4030-ad8f-b21b1bd839b0","Type":"ContainerDied","Data":"c86a854b9bc62e33f778400d227b93a54efc3cbb47307a14958caa3c0cbede75"} Oct 30 05:36:23.522876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:23.522841 6171 generic.go:356] "Generic (PLEG): container finished" podID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerID="73f2c146ca92af5d07e25333de17c2077bbbaa60c4f586339b63397ce2c8befa" exitCode=0 Oct 30 05:36:23.523337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:23.522904 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" event={"ID":"927095aa-0630-4030-ad8f-b21b1bd839b0","Type":"ContainerDied","Data":"73f2c146ca92af5d07e25333de17c2077bbbaa60c4f586339b63397ce2c8befa"} Oct 30 05:36:26.382839 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:36:26.382799 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:36:29.547542 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:29.547473 6171 generic.go:356] "Generic (PLEG): container finished" podID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerID="a57c06497f5dd0cecfc7d325ecb0f2b61f8b96f39f04b1454a7be155f3ca260c" exitCode=0 Oct 30 05:36:29.547872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:29.547559 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" event={"ID":"927095aa-0630-4030-ad8f-b21b1bd839b0","Type":"ContainerDied","Data":"a57c06497f5dd0cecfc7d325ecb0f2b61f8b96f39f04b1454a7be155f3ca260c"} Oct 30 05:36:30.664451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.664429 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:30.858598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.858543 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\") pod \"927095aa-0630-4030-ad8f-b21b1bd839b0\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " Oct 30 05:36:30.858598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.858579 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\" (UniqueName: \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\") pod \"927095aa-0630-4030-ad8f-b21b1bd839b0\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " Oct 30 05:36:30.858745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.858620 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\") pod \"927095aa-0630-4030-ad8f-b21b1bd839b0\" (UID: \"927095aa-0630-4030-ad8f-b21b1bd839b0\") " Oct 30 05:36:30.859181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.859156 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle" pod "927095aa-0630-4030-ad8f-b21b1bd839b0" (UID: "927095aa-0630-4030-ad8f-b21b1bd839b0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:30.860837 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.860819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6" pod "927095aa-0630-4030-ad8f-b21b1bd839b0" (UID: "927095aa-0630-4030-ad8f-b21b1bd839b0"). InnerVolumeSpecName "kube-api-access-b48x6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:30.864669 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.864643 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util" pod "927095aa-0630-4030-ad8f-b21b1bd839b0" (UID: "927095aa-0630-4030-ad8f-b21b1bd839b0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:30.959829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.959808 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b48x6\" (UniqueName: \"kubernetes.io/projected/927095aa-0630-4030-ad8f-b21b1bd839b0-kube-api-access-b48x6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:30.959829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.959827 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:30.959934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:30.959837 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/927095aa-0630-4030-ad8f-b21b1bd839b0-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:31.556093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:31.556039 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" event={"ID":"927095aa-0630-4030-ad8f-b21b1bd839b0","Type":"ContainerDied","Data":"a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718"} Oct 30 05:36:31.556093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:31.556092 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cn87bl" Oct 30 05:36:31.556285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:31.556112 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9bf4fd6db20a033363c5709aefd4b59191355f4e278903700707fad08301718" Oct 30 05:36:38.405342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405308 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr"] Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405564 6171 cpu_manager.go:414] "containerMap: removing container" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="util" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405573 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="util" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405581 6171 cpu_manager.go:414] "containerMap: removing container" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="pull" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405586 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="pull" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405599 6171 cpu_manager.go:414] "containerMap: removing container" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="extract" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405604 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="extract" Oct 30 05:36:38.405702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.405650 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="927095aa-0630-4030-ad8f-b21b1bd839b0" containerName="extract" Oct 30 05:36:38.408464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.408444 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.421237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.421218 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr"] Oct 30 05:36:38.425296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.425277 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 05:36:38.428116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.428093 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:36:38.433203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.433185 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-t42lg\"" Oct 30 05:36:38.506812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.506792 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzx9n\" (UniqueName: \"kubernetes.io/projected/e8e71084-e5c0-4fd2-af5d-0469a9471c69-kube-api-access-dzx9n\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.506956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.506825 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e71084-e5c0-4fd2-af5d-0469a9471c69-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.607801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.607764 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e71084-e5c0-4fd2-af5d-0469a9471c69-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.607888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.607851 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzx9n\" (UniqueName: \"kubernetes.io/projected/e8e71084-e5c0-4fd2-af5d-0469a9471c69-kube-api-access-dzx9n\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.608214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.608192 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e8e71084-e5c0-4fd2-af5d-0469a9471c69-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.633021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.633001 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzx9n\" (UniqueName: \"kubernetes.io/projected/e8e71084-e5c0-4fd2-af5d-0469a9471c69-kube-api-access-dzx9n\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-9hpnr\" (UID: \"e8e71084-e5c0-4fd2-af5d-0469a9471c69\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:38.719238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:38.719189 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" Oct 30 05:36:39.047817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:39.047793 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr"] Oct 30 05:36:39.051015 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:39.050987 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8e71084_e5c0_4fd2_af5d_0469a9471c69.slice/crio-1822759ce2615d1b95e668b6816b2a2836d05ca38403424daceac9ec75c4137f WatchSource:0}: Error finding container 1822759ce2615d1b95e668b6816b2a2836d05ca38403424daceac9ec75c4137f: Status 404 returned error can't find the container with id 1822759ce2615d1b95e668b6816b2a2836d05ca38403424daceac9ec75c4137f Oct 30 05:36:39.582561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:39.582527 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" event={"ID":"e8e71084-e5c0-4fd2-af5d-0469a9471c69","Type":"ContainerStarted","Data":"1822759ce2615d1b95e668b6816b2a2836d05ca38403424daceac9ec75c4137f"} Oct 30 05:36:41.591333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:41.591263 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" event={"ID":"e8e71084-e5c0-4fd2-af5d-0469a9471c69","Type":"ContainerStarted","Data":"65fb5de7db1347497138ad930714d0834f5f396dc2530c780530f3504d9d26a3"} Oct 30 05:36:41.661234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:41.661193 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-9hpnr" podStartSLOduration=2.131443427 podStartE2EDuration="3.661180143s" totalImagesPullingTime="1.529736716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:41.648332666 +0000 UTC m=+260.088709858" watchObservedRunningTime="2026-10-30 05:36:41.661180143 +0000 UTC m=+260.101557334" Oct 30 05:36:42.584677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.584646 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb"] Oct 30 05:36:42.587752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.587737 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.622541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.622513 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-xpj2w\"" Oct 30 05:36:42.622956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.622587 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 05:36:42.623381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.623363 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb"] Oct 30 05:36:42.639535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.639513 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.639622 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.639559 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rcbt\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-kube-api-access-5rcbt\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.713438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.713417 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 05:36:42.740262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.740235 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.740354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.740272 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rcbt\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-kube-api-access-5rcbt\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.763118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.763095 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.763267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.763249 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rcbt\" (UniqueName: \"kubernetes.io/projected/9e53b670-2539-4a61-a1aa-fe5e43005508-kube-api-access-5rcbt\") pod \"cert-manager-webhook-5dbbbf75cc-cdqfb\" (UID: \"9e53b670-2539-4a61-a1aa-fe5e43005508\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.905288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.905217 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp"] Oct 30 05:36:42.911900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.911873 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:42.912012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.911993 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:42.930264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.930244 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 05:36:42.930402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.930340 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 05:36:42.930512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.930495 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-g89vc\"" Oct 30 05:36:42.930918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.930904 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp"] Oct 30 05:36:42.941851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.941831 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:42.941963 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.941867 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:42.942011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:42.941982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.042572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.042546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.042712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.042581 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.042712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.042614 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.042907 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.042891 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.042969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.042934 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.065628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.065602 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.211336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.211271 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb"] Oct 30 05:36:43.214291 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:43.214266 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e53b670_2539_4a61_a1aa_fe5e43005508.slice/crio-abd73c524ddd113fa3dc9b211dfae542111eb74e46b3944380ca6aa6706e91fe WatchSource:0}: Error finding container abd73c524ddd113fa3dc9b211dfae542111eb74e46b3944380ca6aa6706e91fe: Status 404 returned error can't find the container with id abd73c524ddd113fa3dc9b211dfae542111eb74e46b3944380ca6aa6706e91fe Oct 30 05:36:43.240221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.240199 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:43.551416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.551388 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp"] Oct 30 05:36:43.555290 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:43.555259 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9f46433_1165_4680_8a8b_3e6592afb95c.slice/crio-01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c WatchSource:0}: Error finding container 01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c: Status 404 returned error can't find the container with id 01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c Oct 30 05:36:43.602370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.602338 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" event={"ID":"c9f46433-1165-4680-8a8b-3e6592afb95c","Type":"ContainerStarted","Data":"01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c"} Oct 30 05:36:43.603636 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:43.603609 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" event={"ID":"9e53b670-2539-4a61-a1aa-fe5e43005508","Type":"ContainerStarted","Data":"abd73c524ddd113fa3dc9b211dfae542111eb74e46b3944380ca6aa6706e91fe"} Oct 30 05:36:44.610121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:44.610084 6171 generic.go:356] "Generic (PLEG): container finished" podID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerID="2ecb33fea6776eb315093c728636a4d722210d668f25cd29ef326d9fecab9c60" exitCode=0 Oct 30 05:36:44.610582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:44.610146 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" event={"ID":"c9f46433-1165-4680-8a8b-3e6592afb95c","Type":"ContainerDied","Data":"2ecb33fea6776eb315093c728636a4d722210d668f25cd29ef326d9fecab9c60"} Oct 30 05:36:45.620553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:45.620519 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" event={"ID":"9e53b670-2539-4a61-a1aa-fe5e43005508","Type":"ContainerStarted","Data":"63d528229c17c2c1a07d50d2cc36b83d02567fc54b36d45c1089d36ac117d53e"} Oct 30 05:36:45.621037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:45.620592 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:45.663282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:45.663235 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" podStartSLOduration=1.344115725 podStartE2EDuration="3.663216918s" totalImagesPullingTime="2.319101193s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:45.662596578 +0000 UTC m=+264.102973783" watchObservedRunningTime="2026-10-30 05:36:45.663216918 +0000 UTC m=+264.103594108" Oct 30 05:36:46.625974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.625902 6171 generic.go:356] "Generic (PLEG): container finished" podID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerID="465d45aefe65bb540758160a6ede5bc77e0016b894dba153d1f56436bd4b8581" exitCode=0 Oct 30 05:36:46.626352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.625976 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" event={"ID":"c9f46433-1165-4680-8a8b-3e6592afb95c","Type":"ContainerDied","Data":"465d45aefe65bb540758160a6ede5bc77e0016b894dba153d1f56436bd4b8581"} Oct 30 05:36:46.780811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.780779 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw"] Oct 30 05:36:46.783854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.783839 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.802934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.802909 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw"] Oct 30 05:36:46.873255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.873235 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.873376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.873295 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pxxh\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-kube-api-access-6pxxh\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.894025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.893983 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-vq85z\"" Oct 30 05:36:46.974185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.974161 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pxxh\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-kube-api-access-6pxxh\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.974266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.974196 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.997492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.997467 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:46.997771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:46.997752 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pxxh\" (UniqueName: \"kubernetes.io/projected/0dab653c-f160-49e1-a3f7-1fefa5730ebb-kube-api-access-6pxxh\") pod \"cert-manager-cainjector-5c57bbd4fc-mppmw\" (UID: \"0dab653c-f160-49e1-a3f7-1fefa5730ebb\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:47.098270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.098245 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" Oct 30 05:36:47.398623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.398554 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw"] Oct 30 05:36:47.401213 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:47.401188 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dab653c_f160_49e1_a3f7_1fefa5730ebb.slice/crio-42b2a66acd5a8c74151320c7e1bf35ee696b7fc0820e50314baf446b1b288ef4 WatchSource:0}: Error finding container 42b2a66acd5a8c74151320c7e1bf35ee696b7fc0820e50314baf446b1b288ef4: Status 404 returned error can't find the container with id 42b2a66acd5a8c74151320c7e1bf35ee696b7fc0820e50314baf446b1b288ef4 Oct 30 05:36:47.632124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.632081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" event={"ID":"0dab653c-f160-49e1-a3f7-1fefa5730ebb","Type":"ContainerStarted","Data":"ca5cb822da1331b9185cbdf51a938ddcf0bbc3aa38e2d6e292840fa6692c3186"} Oct 30 05:36:47.632562 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.632137 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" event={"ID":"0dab653c-f160-49e1-a3f7-1fefa5730ebb","Type":"ContainerStarted","Data":"42b2a66acd5a8c74151320c7e1bf35ee696b7fc0820e50314baf446b1b288ef4"} Oct 30 05:36:47.634285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.634262 6171 generic.go:356] "Generic (PLEG): container finished" podID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerID="b9dccf663d05478097b1ab21399e49e4fece014e244fec5df3b277406771776a" exitCode=0 Oct 30 05:36:47.634381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.634322 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" event={"ID":"c9f46433-1165-4680-8a8b-3e6592afb95c","Type":"ContainerDied","Data":"b9dccf663d05478097b1ab21399e49e4fece014e244fec5df3b277406771776a"} Oct 30 05:36:47.694796 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:47.694717 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-mppmw" podStartSLOduration=1.694704909 podStartE2EDuration="1.694704909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:47.694485379 +0000 UTC m=+266.134862632" watchObservedRunningTime="2026-10-30 05:36:47.694704909 +0000 UTC m=+266.135082097" Oct 30 05:36:48.762836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.762811 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:48.788557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.788535 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\") pod \"c9f46433-1165-4680-8a8b-3e6592afb95c\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " Oct 30 05:36:48.788689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.788570 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\") pod \"c9f46433-1165-4680-8a8b-3e6592afb95c\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " Oct 30 05:36:48.788689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.788641 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\") pod \"c9f46433-1165-4680-8a8b-3e6592afb95c\" (UID: \"c9f46433-1165-4680-8a8b-3e6592afb95c\") " Oct 30 05:36:48.788961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.788941 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle" pod "c9f46433-1165-4680-8a8b-3e6592afb95c" (UID: "c9f46433-1165-4680-8a8b-3e6592afb95c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:48.790586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.790566 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7" pod "c9f46433-1165-4680-8a8b-3e6592afb95c" (UID: "c9f46433-1165-4680-8a8b-3e6592afb95c"). InnerVolumeSpecName "kube-api-access-6lsb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:36:48.868812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.868783 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util" pod "c9f46433-1165-4680-8a8b-3e6592afb95c" (UID: "c9f46433-1165-4680-8a8b-3e6592afb95c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:36:48.893541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.889567 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:48.893541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.889596 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c9f46433-1165-4680-8a8b-3e6592afb95c-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:48.893541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:48.889661 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lsb7\" (UniqueName: \"kubernetes.io/projected/c9f46433-1165-4680-8a8b-3e6592afb95c-kube-api-access-6lsb7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:36:49.643761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:49.643735 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" Oct 30 05:36:49.643918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:49.643727 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc85kr4pp" event={"ID":"c9f46433-1165-4680-8a8b-3e6592afb95c","Type":"ContainerDied","Data":"01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c"} Oct 30 05:36:49.643918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:49.643874 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01699572ad10d313174be7722ff2ca25e226a3455d27041f34a996c14d838b2c" Oct 30 05:36:51.628230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:51.628195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" Oct 30 05:36:56.114889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.114854 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl"] Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115202 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="extract" Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115214 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="extract" Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115228 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="util" Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115232 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="util" Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115238 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="pull" Oct 30 05:36:56.115292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115243 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="pull" Oct 30 05:36:56.115472 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.115297 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9f46433-1165-4680-8a8b-3e6592afb95c" containerName="extract" Oct 30 05:36:56.160970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.160944 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl"] Oct 30 05:36:56.161114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.161026 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.175887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.175869 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:36:56.175987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.175892 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 05:36:56.187893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.187875 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xt4qz\"" Oct 30 05:36:56.243768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.243748 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-tmp\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.243854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.243797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98fw7\" (UniqueName: \"kubernetes.io/projected/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-kube-api-access-98fw7\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.344648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.344625 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98fw7\" (UniqueName: \"kubernetes.io/projected/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-kube-api-access-98fw7\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.344734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.344665 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-tmp\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.344967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.344951 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-tmp\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.369128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.369075 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98fw7\" (UniqueName: \"kubernetes.io/projected/1b50f40b-3b6b-4032-aa09-9a55ac8c4f51-kube-api-access-98fw7\") pod \"openshift-lws-operator-8465994787-6hhgl\" (UID: \"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.379614 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:36:56.379588 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:36:56.484112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.484088 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" Oct 30 05:36:56.790355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:56.790326 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl"] Oct 30 05:36:56.793517 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:36:56.793486 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b50f40b_3b6b_4032_aa09_9a55ac8c4f51.slice/crio-74d0eb025175ea5e5acc954d4f18a86be3f9a3c1dd5a654ccfba524f346e4f9c WatchSource:0}: Error finding container 74d0eb025175ea5e5acc954d4f18a86be3f9a3c1dd5a654ccfba524f346e4f9c: Status 404 returned error can't find the container with id 74d0eb025175ea5e5acc954d4f18a86be3f9a3c1dd5a654ccfba524f346e4f9c Oct 30 05:36:57.673817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:57.673784 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" event={"ID":"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51","Type":"ContainerStarted","Data":"74d0eb025175ea5e5acc954d4f18a86be3f9a3c1dd5a654ccfba524f346e4f9c"} Oct 30 05:36:59.683650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:36:59.683614 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" event={"ID":"1b50f40b-3b6b-4032-aa09-9a55ac8c4f51","Type":"ContainerStarted","Data":"c174b8c7233e444a4e4bed6a0db52285e3024747ac74ca7e1046b061602e192b"} Oct 30 05:37:02.531842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.531743 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-6hhgl" podStartSLOduration=3.889983523 podStartE2EDuration="6.531728314s" totalImagesPullingTime="2.641744791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:59.724648578 +0000 UTC m=+278.165025771" watchObservedRunningTime="2026-10-30 05:37:02.531728314 +0000 UTC m=+280.972105505" Oct 30 05:37:02.532242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.532166 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l"] Oct 30 05:37:02.535456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.535441 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.550318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.550299 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 05:37:02.562452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.562420 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 05:37:02.562579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.562501 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-9tfcx\"" Oct 30 05:37:02.563515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.563500 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l"] Oct 30 05:37:02.594624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.594579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.594772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.594637 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fbp\" (UniqueName: \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.594772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.594758 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.695630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.695597 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.695801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.695647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.695801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.695688 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fbp\" (UniqueName: \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.695965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.695947 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.696009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.695989 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.719572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.719547 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fbp\" (UniqueName: \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:02.850885 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:02.850798 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:03.152500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:03.152477 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l"] Oct 30 05:37:03.154875 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:03.154849 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4181a0b2_ce22_4f06_8973_ff2064c8cd34.slice/crio-c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae WatchSource:0}: Error finding container c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae: Status 404 returned error can't find the container with id c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae Oct 30 05:37:03.700551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:03.700513 6171 generic.go:356] "Generic (PLEG): container finished" podID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerID="1c90d916e4cdf751638202e69331d89c647ab0bf9ce72a3d15b6bb46186e43f8" exitCode=0 Oct 30 05:37:03.700929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:03.700552 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" event={"ID":"4181a0b2-ce22-4f06-8973-ff2064c8cd34","Type":"ContainerDied","Data":"1c90d916e4cdf751638202e69331d89c647ab0bf9ce72a3d15b6bb46186e43f8"} Oct 30 05:37:03.700929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:03.700595 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" event={"ID":"4181a0b2-ce22-4f06-8973-ff2064c8cd34","Type":"ContainerStarted","Data":"c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae"} Oct 30 05:37:04.706369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:04.706287 6171 generic.go:356] "Generic (PLEG): container finished" podID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerID="5a26c8cb787f38b99592f5eada7670623d26569e5a4439cc0d766ba06d4a4716" exitCode=0 Oct 30 05:37:04.706369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:04.706346 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" event={"ID":"4181a0b2-ce22-4f06-8973-ff2064c8cd34","Type":"ContainerDied","Data":"5a26c8cb787f38b99592f5eada7670623d26569e5a4439cc0d766ba06d4a4716"} Oct 30 05:37:05.712394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:05.712360 6171 generic.go:356] "Generic (PLEG): container finished" podID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerID="5fb1e7d16994148f03b943d67af1852577210523c106b2d898126a09224625b8" exitCode=0 Oct 30 05:37:05.712769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:05.712412 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" event={"ID":"4181a0b2-ce22-4f06-8973-ff2064c8cd34","Type":"ContainerDied","Data":"5fb1e7d16994148f03b943d67af1852577210523c106b2d898126a09224625b8"} Oct 30 05:37:06.839943 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.839918 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:06.927333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.927305 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\" (UniqueName: \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\") pod \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " Oct 30 05:37:06.927480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.927339 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\") pod \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " Oct 30 05:37:06.927480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.927399 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\") pod \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\" (UID: \"4181a0b2-ce22-4f06-8973-ff2064c8cd34\") " Oct 30 05:37:06.928001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.927934 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle" pod "4181a0b2-ce22-4f06-8973-ff2064c8cd34" (UID: "4181a0b2-ce22-4f06-8973-ff2064c8cd34"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:06.929523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.929501 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp" pod "4181a0b2-ce22-4f06-8973-ff2064c8cd34" (UID: "4181a0b2-ce22-4f06-8973-ff2064c8cd34"). InnerVolumeSpecName "kube-api-access-d7fbp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:37:06.934947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:06.934920 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util" pod "4181a0b2-ce22-4f06-8973-ff2064c8cd34" (UID: "4181a0b2-ce22-4f06-8973-ff2064c8cd34"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:37:07.028445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.028425 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:37:07.028525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.028445 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fbp\" (UniqueName: \"kubernetes.io/projected/4181a0b2-ce22-4f06-8973-ff2064c8cd34-kube-api-access-d7fbp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:37:07.028525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.028455 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4181a0b2-ce22-4f06-8973-ff2064c8cd34-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:37:07.722537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.722512 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" Oct 30 05:37:07.722689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.722511 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mcn4l" event={"ID":"4181a0b2-ce22-4f06-8973-ff2064c8cd34","Type":"ContainerDied","Data":"c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae"} Oct 30 05:37:07.722689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:07.722633 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c76237764b83af5e14f58fb88e7824dc8ac20e7af85ce6799e20a320307613ae" Oct 30 05:37:17.886761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.886730 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv"] Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.886993 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="extract" Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887003 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="extract" Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887025 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="util" Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887029 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="util" Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887094 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="pull" Oct 30 05:37:17.887128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887101 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="pull" Oct 30 05:37:17.887315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.887163 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4181a0b2-ce22-4f06-8973-ff2064c8cd34" containerName="extract" Oct 30 05:37:17.890431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.890416 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:17.907369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907344 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 05:37:17.907470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907410 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 05:37:17.907618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907603 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 05:37:17.907655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907636 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 05:37:17.907655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907645 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 05:37:17.907717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.907654 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dmsd5\"" Oct 30 05:37:17.929693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:17.929663 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv"] Oct 30 05:37:18.006328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006304 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.006448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006331 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.006448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006359 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.006448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006414 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.006625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006503 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqgh4\" (UniqueName: \"kubernetes.io/projected/465f403f-78c3-4195-9fbd-cd25c839fb2e-kube-api-access-rqgh4\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.006625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.006592 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107512 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107549 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107568 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqgh4\" (UniqueName: \"kubernetes.io/projected/465f403f-78c3-4195-9fbd-cd25c839fb2e-kube-api-access-rqgh4\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107588 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.107649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.107638 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.108397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.108371 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-manifests\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.114534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.110852 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.114534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.111236 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-webhook-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.114534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.111308 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/465f403f-78c3-4195-9fbd-cd25c839fb2e-tmp\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.114534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.111511 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/465f403f-78c3-4195-9fbd-cd25c839fb2e-metrics-cert\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.157496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.157450 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqgh4\" (UniqueName: \"kubernetes.io/projected/465f403f-78c3-4195-9fbd-cd25c839fb2e-kube-api-access-rqgh4\") pod \"opendatahub-operator-controller-manager-f7d897f7-z5lzv\" (UID: \"465f403f-78c3-4195-9fbd-cd25c839fb2e\") " pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.204414 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.204391 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:18.625083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.625036 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv"] Oct 30 05:37:18.627444 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:18.627418 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod465f403f_78c3_4195_9fbd_cd25c839fb2e.slice/crio-377d0aadbe127a5f20432a9e4f874efc3a344cda204b577da3854db7fa84e303 WatchSource:0}: Error finding container 377d0aadbe127a5f20432a9e4f874efc3a344cda204b577da3854db7fa84e303: Status 404 returned error can't find the container with id 377d0aadbe127a5f20432a9e4f874efc3a344cda204b577da3854db7fa84e303 Oct 30 05:37:18.765777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:18.765746 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" event={"ID":"465f403f-78c3-4195-9fbd-cd25c839fb2e","Type":"ContainerStarted","Data":"377d0aadbe127a5f20432a9e4f874efc3a344cda204b577da3854db7fa84e303"} Oct 30 05:37:21.784026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:21.783988 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" event={"ID":"465f403f-78c3-4195-9fbd-cd25c839fb2e","Type":"ContainerStarted","Data":"c8ebda9e9eb53fbe965f4b4aa9a0f341ded9863ade40c635cd923436115b65a2"} Oct 30 05:37:22.789552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:22.789518 6171 generic.go:356] "Generic (PLEG): container finished" podID="465f403f-78c3-4195-9fbd-cd25c839fb2e" containerID="c8ebda9e9eb53fbe965f4b4aa9a0f341ded9863ade40c635cd923436115b65a2" exitCode=0 Oct 30 05:37:22.789983 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:22.789598 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" event={"ID":"465f403f-78c3-4195-9fbd-cd25c839fb2e","Type":"ContainerDied","Data":"c8ebda9e9eb53fbe965f4b4aa9a0f341ded9863ade40c635cd923436115b65a2"} Oct 30 05:37:23.795228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:23.795193 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" event={"ID":"465f403f-78c3-4195-9fbd-cd25c839fb2e","Type":"ContainerStarted","Data":"a7d28aa322dd7cad863eeaf4763e66a7279c6472aacadb083c5e7d1fa42108ee"} Oct 30 05:37:23.795617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:23.795305 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:23.922481 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:23.922429 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" podStartSLOduration=3.540340842 podStartE2EDuration="6.922416355s" totalImagesPullingTime="2.910177481s" totalInitContainerRuntime="471.898032ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:23.909507964 +0000 UTC m=+302.349885155" watchObservedRunningTime="2026-10-30 05:37:23.922416355 +0000 UTC m=+302.362793547" Oct 30 05:37:26.377772 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:37:26.377738 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:37:34.801614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:34.801583 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-f7d897f7-z5lzv" Oct 30 05:37:37.732263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.732226 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-7k748"] Oct 30 05:37:37.736792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.736773 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.755132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.755107 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 05:37:37.755132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.755119 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:37:37.755369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.755355 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:37:37.767033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.767013 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 05:37:37.767301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.767286 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-p9n4d\"" Oct 30 05:37:37.809007 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.808976 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-7k748"] Oct 30 05:37:37.851373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.851342 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jczwl\" (UniqueName: \"kubernetes.io/projected/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-kube-api-access-jczwl\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.851530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.851387 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tls-certs\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.851530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.851500 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tmp\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.952238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.952204 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tmp\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.952434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.952272 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jczwl\" (UniqueName: \"kubernetes.io/projected/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-kube-api-access-jczwl\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.952434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.952309 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tls-certs\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.954553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.954532 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tmp\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.954759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.954743 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-tls-certs\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:37.985857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:37.985798 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jczwl\" (UniqueName: \"kubernetes.io/projected/d8d4e9ee-4fee-4c4b-8187-e7a44b20e101-kube-api-access-jczwl\") pod \"kube-auth-proxy-8596b47b87-7k748\" (UID: \"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101\") " pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:38.049271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:38.049241 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" Oct 30 05:37:38.349615 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:38.349543 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8596b47b87-7k748"] Oct 30 05:37:38.352465 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:38.352433 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8d4e9ee_4fee_4c4b_8187_e7a44b20e101.slice/crio-fd87bc19b499a78c37d9de0988d71f5e35a8df69f9be94462a69a85083bc212b WatchSource:0}: Error finding container fd87bc19b499a78c37d9de0988d71f5e35a8df69f9be94462a69a85083bc212b: Status 404 returned error can't find the container with id fd87bc19b499a78c37d9de0988d71f5e35a8df69f9be94462a69a85083bc212b Oct 30 05:37:38.854759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:38.854719 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" event={"ID":"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101","Type":"ContainerStarted","Data":"fd87bc19b499a78c37d9de0988d71f5e35a8df69f9be94462a69a85083bc212b"} Oct 30 05:37:39.540897 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.540867 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:37:39.544631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.544602 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.566555 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.566533 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 05:37:39.566992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.566931 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 05:37:39.567173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.567017 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 05:37:39.567252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.567236 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 05:37:39.578726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.578687 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-7ckgr\"" Oct 30 05:37:39.581264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.581240 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669129 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669174 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv97v\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669279 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669312 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669364 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669395 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.669490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.669418 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.770638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770602 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.770826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770662 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.770826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770764 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.770826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770807 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.770826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770826 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.771032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770846 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.771032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.770862 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv97v\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.771591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.771562 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.773425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.773401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.773533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.773497 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.773605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.773552 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.774067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.774026 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.794945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.794891 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv97v\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.795082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.794990 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-fcmfp\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:39.857996 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:39.857973 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:40.187366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:40.187295 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:37:40.191990 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:40.191963 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6d8768_b8b6_4b08_829a_0aa537367326.slice/crio-c2fea2287e0a4251d0a0c28ff5a30dbf1b237cd3302f2eb87304bb67d06a2b01 WatchSource:0}: Error finding container c2fea2287e0a4251d0a0c28ff5a30dbf1b237cd3302f2eb87304bb67d06a2b01: Status 404 returned error can't find the container with id c2fea2287e0a4251d0a0c28ff5a30dbf1b237cd3302f2eb87304bb67d06a2b01 Oct 30 05:37:40.864492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:40.864411 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" event={"ID":"be6d8768-b8b6-4b08-829a-0aa537367326","Type":"ContainerStarted","Data":"c2fea2287e0a4251d0a0c28ff5a30dbf1b237cd3302f2eb87304bb67d06a2b01"} Oct 30 05:37:40.866444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:40.866413 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" event={"ID":"d8d4e9ee-4fee-4c4b-8187-e7a44b20e101","Type":"ContainerStarted","Data":"7a9aba371b87b57382f5ddf8691b55a1842e4ce0634098e19bf683e613d3a194"} Oct 30 05:37:40.916994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:40.916946 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8596b47b87-7k748" podStartSLOduration=1.66500248 podStartE2EDuration="3.916932185s" totalImagesPullingTime="2.251929705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:40.915677738 +0000 UTC m=+319.356054923" watchObservedRunningTime="2026-10-30 05:37:40.916932185 +0000 UTC m=+319.357309377" Oct 30 05:37:41.624755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.624725 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-7nfbb"] Oct 30 05:37:41.631587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.631562 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:41.650206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.650182 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-7nfbb"] Oct 30 05:37:41.660001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.659971 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 05:37:41.660206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.660187 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-qgjrl\"" Oct 30 05:37:41.790559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.790526 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sll2b\" (UniqueName: \"kubernetes.io/projected/b1f10f8e-96a4-4c21-9175-31da2899dee8-kube-api-access-sll2b\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:41.790850 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.790832 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:41.892212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.892128 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sll2b\" (UniqueName: \"kubernetes.io/projected/b1f10f8e-96a4-4c21-9175-31da2899dee8-kube-api-access-sll2b\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:41.892710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.892239 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:41.892710 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:37:41.892408 6171 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 05:37:41.892710 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:37:41.892465 6171 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs podName:b1f10f8e-96a4-4c21-9175-31da2899dee8 nodeName:}" failed. No retries permitted until 2026-10-30 05:37:42.392448034 +0000 UTC m=+320.832825217 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs") pod "odh-observability-584f99c788-7nfbb" (UID: "b1f10f8e-96a4-4c21-9175-31da2899dee8") : secret "odh-observability-webhook-cert" not found Oct 30 05:37:41.916468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:41.916442 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sll2b\" (UniqueName: \"kubernetes.io/projected/b1f10f8e-96a4-4c21-9175-31da2899dee8-kube-api-access-sll2b\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:42.397235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:42.397196 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:42.400101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:42.400079 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/b1f10f8e-96a4-4c21-9175-31da2899dee8-webhook-certs\") pod \"odh-observability-584f99c788-7nfbb\" (UID: \"b1f10f8e-96a4-4c21-9175-31da2899dee8\") " pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:42.545767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:42.545731 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:42.870289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:42.870260 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-7nfbb"] Oct 30 05:37:42.872413 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:42.872382 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1f10f8e_96a4_4c21_9175_31da2899dee8.slice/crio-e5f707ae48c0cdfdfc68a3699c65baa8fa31cc8dfa4e215c3bcc9a9c11581f2e WatchSource:0}: Error finding container e5f707ae48c0cdfdfc68a3699c65baa8fa31cc8dfa4e215c3bcc9a9c11581f2e: Status 404 returned error can't find the container with id e5f707ae48c0cdfdfc68a3699c65baa8fa31cc8dfa4e215c3bcc9a9c11581f2e Oct 30 05:37:43.019320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.019284 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:37:43.019714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.019357 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:37:43.883856 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.883750 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" event={"ID":"be6d8768-b8b6-4b08-829a-0aa537367326","Type":"ContainerStarted","Data":"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817"} Oct 30 05:37:43.883856 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.883815 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:43.885172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.885148 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-7nfbb" event={"ID":"b1f10f8e-96a4-4c21-9175-31da2899dee8","Type":"ContainerStarted","Data":"e5f707ae48c0cdfdfc68a3699c65baa8fa31cc8dfa4e215c3bcc9a9c11581f2e"} Oct 30 05:37:43.954332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:43.954275 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" podStartSLOduration=2.130178525 podStartE2EDuration="4.954256843s" totalImagesPullingTime="2.824078318s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:43.94133655 +0000 UTC m=+322.381713740" watchObservedRunningTime="2026-10-30 05:37:43.954256843 +0000 UTC m=+322.394634035" Oct 30 05:37:44.892298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:44.892269 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:37:45.895927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:45.895894 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-7nfbb" event={"ID":"b1f10f8e-96a4-4c21-9175-31da2899dee8","Type":"ContainerStarted","Data":"97850ff9e7e647bb8277838ae39c7e5f57926eb15dfbbf8f12c2979fdc93de5b"} Oct 30 05:37:45.936144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:45.936098 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-7nfbb" podStartSLOduration=2.829787633 podStartE2EDuration="4.936085196s" totalImagesPullingTime="2.106297563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:45.935168365 +0000 UTC m=+324.375545554" watchObservedRunningTime="2026-10-30 05:37:45.936085196 +0000 UTC m=+324.376462389" Oct 30 05:37:46.900524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:46.900493 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:55.255667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.255633 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm"] Oct 30 05:37:55.260833 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.260813 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.276067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.276029 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-n722x\"" Oct 30 05:37:55.303532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.303504 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.303645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.303550 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4f7z\" (UniqueName: \"kubernetes.io/projected/55458753-8f72-4e41-b495-6ce7fda91c10-kube-api-access-k4f7z\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.303684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.303647 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.303684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.303663 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55458753-8f72-4e41-b495-6ce7fda91c10-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.312993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.312969 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Oct 30 05:37:55.314239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.314221 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm"] Oct 30 05:37:55.403989 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.403965 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404002 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4f7z\" (UniqueName: \"kubernetes.io/projected/55458753-8f72-4e41-b495-6ce7fda91c10-kube-api-access-k4f7z\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404036 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404070 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55458753-8f72-4e41-b495-6ce7fda91c10-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404323 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-tmp\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404409 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55458753-8f72-4e41-b495-6ce7fda91c10-manifests\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.404827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.404812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/55458753-8f72-4e41-b495-6ce7fda91c10-trusted-ca\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.429445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.429422 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4f7z\" (UniqueName: \"kubernetes.io/projected/55458753-8f72-4e41-b495-6ce7fda91c10-kube-api-access-k4f7z\") pod \"kserve-module-controller-manager-5cfb77bf8c-vghzm\" (UID: \"55458753-8f72-4e41-b495-6ce7fda91c10\") " pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.572095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.572025 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:55.887013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.886949 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm"] Oct 30 05:37:55.889956 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:37:55.889928 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55458753_8f72_4e41_b495_6ce7fda91c10.slice/crio-f330dd3077230d77e7d09fbf3de2b7cf94da1aa91b686f80e0e4c774e56d91e3 WatchSource:0}: Error finding container f330dd3077230d77e7d09fbf3de2b7cf94da1aa91b686f80e0e4c774e56d91e3: Status 404 returned error can't find the container with id f330dd3077230d77e7d09fbf3de2b7cf94da1aa91b686f80e0e4c774e56d91e3 Oct 30 05:37:55.891648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.891633 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:37:55.939022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:55.938996 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" event={"ID":"55458753-8f72-4e41-b495-6ce7fda91c10","Type":"ContainerStarted","Data":"f330dd3077230d77e7d09fbf3de2b7cf94da1aa91b686f80e0e4c774e56d91e3"} Oct 30 05:37:56.381105 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:37:56.381066 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:37:57.906656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:57.906631 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-7nfbb" Oct 30 05:37:57.949792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:57.949756 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" event={"ID":"55458753-8f72-4e41-b495-6ce7fda91c10","Type":"ContainerStarted","Data":"7b897ec1e57caf0fccef3ab4d13cac2dc8b4bafb3c953f9238a2a85bf08b16ae"} Oct 30 05:37:57.949953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:57.949809 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:37:57.994280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:37:57.994229 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" podStartSLOduration=1.6033037110000001 podStartE2EDuration="2.994216479s" totalImagesPullingTime="1.390912768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:37:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:37:57.993854 +0000 UTC m=+336.434231191" watchObservedRunningTime="2026-10-30 05:37:57.994216479 +0000 UTC m=+336.434593670" Oct 30 05:38:18.980461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:18.980426 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-5cfb77bf8c-vghzm" Oct 30 05:38:26.379582 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:38:26.379547 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:38:41.103787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.103715 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq"] Oct 30 05:38:41.110301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.110278 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.135746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.135721 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq"] Oct 30 05:38:41.262667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262638 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7285465-20bc-47c8-82dc-b4baa7f4b38b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.262838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262674 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.262838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262757 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.262838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262799 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.262948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262860 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.262948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262885 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xksrl\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-kube-api-access-xksrl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.263014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.262953 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.363992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.363921 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.363992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.363956 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.363992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.363978 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.364256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.364007 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xksrl\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-kube-api-access-xksrl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.364256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.364076 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.364256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.364127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7285465-20bc-47c8-82dc-b4baa7f4b38b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.364256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.364153 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.364792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.364723 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.366488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.366465 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7285465-20bc-47c8-82dc-b4baa7f4b38b-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.366660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.366643 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.366922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.366900 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.366963 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.366916 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.387484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.387465 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.387960 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.387938 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xksrl\" (UniqueName: \"kubernetes.io/projected/e7285465-20bc-47c8-82dc-b4baa7f4b38b-kube-api-access-xksrl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-rdsvq\" (UID: \"e7285465-20bc-47c8-82dc-b4baa7f4b38b\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.420944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.420920 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:41.744594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.744376 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq"] Oct 30 05:38:41.746782 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:38:41.746753 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7285465_20bc_47c8_82dc_b4baa7f4b38b.slice/crio-8a91f813898334c7051152e36be592adabff7fc2f968494b478b0086c95a7004 WatchSource:0}: Error finding container 8a91f813898334c7051152e36be592adabff7fc2f968494b478b0086c95a7004: Status 404 returned error can't find the container with id 8a91f813898334c7051152e36be592adabff7fc2f968494b478b0086c95a7004 Oct 30 05:38:41.749070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.749020 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:38:41.749177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:41.749095 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:38:42.135427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.135340 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" event={"ID":"e7285465-20bc-47c8-82dc-b4baa7f4b38b","Type":"ContainerStarted","Data":"0411028c12be9fe6328fafdda15c810fa4ed0f59329bbb95e6044ad3ed892f5d"} Oct 30 05:38:42.135427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.135394 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" event={"ID":"e7285465-20bc-47c8-82dc-b4baa7f4b38b","Type":"ContainerStarted","Data":"8a91f813898334c7051152e36be592adabff7fc2f968494b478b0086c95a7004"} Oct 30 05:38:42.135919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.135533 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:42.137060 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.137015 6171 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-rdsvq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:38:42.137169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.137084 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" podUID="e7285465-20bc-47c8-82dc-b4baa7f4b38b" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:38:42.200644 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:42.200594 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" podStartSLOduration=1.2005821939999999 podStartE2EDuration="1.200582194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:38:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:38:42.200107233 +0000 UTC m=+380.640484425" watchObservedRunningTime="2026-10-30 05:38:42.200582194 +0000 UTC m=+380.640959385" Oct 30 05:38:43.140320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.140295 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-rdsvq" Oct 30 05:38:43.241309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.241279 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:38:43.241493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.241472 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" podUID="be6d8768-b8b6-4b08-829a-0aa537367326" containerName="discovery" containerID="cri-o://2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817" gracePeriod=30 Oct 30 05:38:43.479077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.479035 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:38:43.582785 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582755 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.582935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582803 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.582935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582846 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.582935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582864 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.582935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582886 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.582935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582925 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.583210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.582966 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\") pod \"be6d8768-b8b6-4b08-829a-0aa537367326\" (UID: \"be6d8768-b8b6-4b08-829a-0aa537367326\") " Oct 30 05:38:43.583368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.583337 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:38:43.585400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:43.585517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585458 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:43.585629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585538 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:38:43.585629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585537 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:38:43.585709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "kube-api-access-dv97v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.585709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.585688 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token" pod "be6d8768-b8b6-4b08-829a-0aa537367326" (UID: "be6d8768-b8b6-4b08-829a-0aa537367326"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:38:43.684216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684197 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684215 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv97v\" (UniqueName: \"kubernetes.io/projected/be6d8768-b8b6-4b08-829a-0aa537367326-kube-api-access-dv97v\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684224 6171 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-cacerts\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684232 6171 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-kubeconfig\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684240 6171 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/be6d8768-b8b6-4b08-829a-0aa537367326-local-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684247 6171 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-dns-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:43.684351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:43.684255 6171 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/be6d8768-b8b6-4b08-829a-0aa537367326-istio-csr-ca-configmap\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:38:44.145334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.145302 6171 generic.go:356] "Generic (PLEG): container finished" podID="be6d8768-b8b6-4b08-829a-0aa537367326" containerID="2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817" exitCode=0 Oct 30 05:38:44.145718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.145348 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" Oct 30 05:38:44.145718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.145389 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" event={"ID":"be6d8768-b8b6-4b08-829a-0aa537367326","Type":"ContainerDied","Data":"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817"} Oct 30 05:38:44.145718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.145439 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp" event={"ID":"be6d8768-b8b6-4b08-829a-0aa537367326","Type":"ContainerDied","Data":"c2fea2287e0a4251d0a0c28ff5a30dbf1b237cd3302f2eb87304bb67d06a2b01"} Oct 30 05:38:44.145718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.145460 6171 scope.go:122] "RemoveContainer" containerID="2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817" Oct 30 05:38:44.153646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.153629 6171 scope.go:122] "RemoveContainer" containerID="2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817" Oct 30 05:38:44.153894 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:38:44.153877 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817\": container with ID starting with 2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817 not found: ID does not exist" containerID="2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817" Oct 30 05:38:44.153941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.153902 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817"} err="failed to get container status \"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817\": rpc error: code = NotFound desc = could not find container \"2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817\": container with ID starting with 2dbe99f4b115c912da76a2a599b3d7aa5e61eecb8a616479bce8016a00f82817 not found: ID does not exist" Oct 30 05:38:44.196794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.196768 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:38:44.208954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:44.208926 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-fcmfp"] Oct 30 05:38:45.668586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:38:45.668554 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6d8768-b8b6-4b08-829a-0aa537367326" path="/var/lib/kubelet/pods/be6d8768-b8b6-4b08-829a-0aa537367326/volumes" Oct 30 05:38:56.378418 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:38:56.378379 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:39:26.377747 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:39:26.377711 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:39:33.601230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.601195 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv"] Oct 30 05:39:33.601814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.601654 6171 cpu_manager.go:414] "containerMap: removing container" podUID="be6d8768-b8b6-4b08-829a-0aa537367326" containerName="discovery" Oct 30 05:39:33.601814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.601671 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6d8768-b8b6-4b08-829a-0aa537367326" containerName="discovery" Oct 30 05:39:33.601814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.601789 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6d8768-b8b6-4b08-829a-0aa537367326" containerName="discovery" Oct 30 05:39:33.604930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.604909 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.635300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.635268 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 05:39:33.635300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.635267 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 05:39:33.635300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.635293 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-7thgl\"" Oct 30 05:39:33.636386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.636368 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv"] Oct 30 05:39:33.681902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.681880 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.681902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.681903 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.682128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.681982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g457\" (UniqueName: \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.698822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.698800 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5"] Oct 30 05:39:33.702021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.702006 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.719416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.719395 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-mpb6v\"" Oct 30 05:39:33.723827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.723805 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5"] Oct 30 05:39:33.782916 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.782888 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.783035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.782934 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.783035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.782949 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.783035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.782969 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48ssf\" (UniqueName: \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.783196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.783111 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.783196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.783182 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g457\" (UniqueName: \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.783360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.783341 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.783397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.783352 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.806383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.806363 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf"] Oct 30 05:39:33.809845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.809826 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.810524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.810503 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g457\" (UniqueName: \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.834671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.834644 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf"] Oct 30 05:39:33.837440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.837422 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-jtm6g\"" Oct 30 05:39:33.884135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884076 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.884237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884132 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.884237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884169 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.884237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jngcc\" (UniqueName: \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.884237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884225 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48ssf\" (UniqueName: \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.884463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884297 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.884463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884457 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.884534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.884521 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.894962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.894941 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52"] Oct 30 05:39:33.898385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.898370 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:33.908437 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.908420 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48ssf\" (UniqueName: \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:33.919456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.919436 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52"] Oct 30 05:39:33.920398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.920380 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:33.937624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.937608 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-xk7kf\"" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985516 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985565 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985583 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jngcc\" (UniqueName: \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985602 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qj2k\" (UniqueName: \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985631 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.985658 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.986013 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:33.987063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:33.986318 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:34.009577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.009554 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jngcc\" (UniqueName: \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:34.020484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.020464 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:34.086638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.086609 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.086769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.086650 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qj2k\" (UniqueName: \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.086769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.086695 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.087038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.087023 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.087038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.087033 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.110216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.110197 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qj2k\" (UniqueName: \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.131295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.131275 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:34.213734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.213708 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:34.219990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.219972 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv"] Oct 30 05:39:34.223036 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:39:34.223011 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a3b6b7c_0a10_455c_a51c_b18368e1008c.slice/crio-b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2 WatchSource:0}: Error finding container b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2: Status 404 returned error can't find the container with id b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2 Oct 30 05:39:34.328563 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.328451 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5"] Oct 30 05:39:34.330619 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:39:34.330591 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f11bf76_b67a_49cf_84dd_b5a43dd0b9d8.slice/crio-8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9 WatchSource:0}: Error finding container 8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9: Status 404 returned error can't find the container with id 8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9 Oct 30 05:39:34.353246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.353222 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" event={"ID":"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8","Type":"ContainerStarted","Data":"8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9"} Oct 30 05:39:34.354625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.354601 6171 generic.go:356] "Generic (PLEG): container finished" podID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerID="6adf3766c3cc6e4d3596e0b7a22825b5c059fbb9589675730bb1e776de0b4118" exitCode=0 Oct 30 05:39:34.354707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.354634 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" event={"ID":"6a3b6b7c-0a10-455c-a51c-b18368e1008c","Type":"ContainerDied","Data":"6adf3766c3cc6e4d3596e0b7a22825b5c059fbb9589675730bb1e776de0b4118"} Oct 30 05:39:34.354707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.354663 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" event={"ID":"6a3b6b7c-0a10-455c-a51c-b18368e1008c","Type":"ContainerStarted","Data":"b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2"} Oct 30 05:39:34.453030 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.452981 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf"] Oct 30 05:39:34.455645 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:39:34.455614 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28dd250_e472_4d5c_b1f9_d747d40edca0.slice/crio-00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34 WatchSource:0}: Error finding container 00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34: Status 404 returned error can't find the container with id 00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34 Oct 30 05:39:34.517706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:34.517688 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52"] Oct 30 05:39:34.545295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:39:34.545272 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f140c6_30e4_4153_ae27_15d82a63021e.slice/crio-0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7 WatchSource:0}: Error finding container 0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7: Status 404 returned error can't find the container with id 0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7 Oct 30 05:39:35.361220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.361140 6171 generic.go:356] "Generic (PLEG): container finished" podID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerID="97cd0f716d4c081a35fbe31fe6fe95abd0b103ca1e83b727d64ffb4690e8dae8" exitCode=0 Oct 30 05:39:35.361601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.361225 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" event={"ID":"6a3b6b7c-0a10-455c-a51c-b18368e1008c","Type":"ContainerDied","Data":"97cd0f716d4c081a35fbe31fe6fe95abd0b103ca1e83b727d64ffb4690e8dae8"} Oct 30 05:39:35.362860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.362838 6171 generic.go:356] "Generic (PLEG): container finished" podID="74f140c6-30e4-4153-ae27-15d82a63021e" containerID="c86534404cd3d87885b001dde8442be0ee863a882d248ecf5593c4079e68acd6" exitCode=0 Oct 30 05:39:35.362917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.362904 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" event={"ID":"74f140c6-30e4-4153-ae27-15d82a63021e","Type":"ContainerDied","Data":"c86534404cd3d87885b001dde8442be0ee863a882d248ecf5593c4079e68acd6"} Oct 30 05:39:35.362969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.362937 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" event={"ID":"74f140c6-30e4-4153-ae27-15d82a63021e","Type":"ContainerStarted","Data":"0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7"} Oct 30 05:39:35.364571 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.364545 6171 generic.go:356] "Generic (PLEG): container finished" podID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerID="4379de47a91432f0e030c3bbfa9e07881bb2666b49e4fd77d86e98d67f7f5df9" exitCode=0 Oct 30 05:39:35.364673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.364631 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" event={"ID":"f28dd250-e472-4d5c-b1f9-d747d40edca0","Type":"ContainerDied","Data":"4379de47a91432f0e030c3bbfa9e07881bb2666b49e4fd77d86e98d67f7f5df9"} Oct 30 05:39:35.364840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.364675 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" event={"ID":"f28dd250-e472-4d5c-b1f9-d747d40edca0","Type":"ContainerStarted","Data":"00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34"} Oct 30 05:39:35.366318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.366294 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerID="fbcbf95d838c9a16c7a1a47aaaca04e21c8f9961e47914d7fdf2a465167f6b53" exitCode=0 Oct 30 05:39:35.366427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:35.366321 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" event={"ID":"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8","Type":"ContainerDied","Data":"fbcbf95d838c9a16c7a1a47aaaca04e21c8f9961e47914d7fdf2a465167f6b53"} Oct 30 05:39:36.372552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:36.372486 6171 generic.go:356] "Generic (PLEG): container finished" podID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerID="ca602a7f3cfd34ede27b324cd9a151b347e9f892ddf776aee15f67bf5d812227" exitCode=0 Oct 30 05:39:36.372912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:36.372563 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" event={"ID":"6a3b6b7c-0a10-455c-a51c-b18368e1008c","Type":"ContainerDied","Data":"ca602a7f3cfd34ede27b324cd9a151b347e9f892ddf776aee15f67bf5d812227"} Oct 30 05:39:36.374241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:36.374217 6171 generic.go:356] "Generic (PLEG): container finished" podID="74f140c6-30e4-4153-ae27-15d82a63021e" containerID="f7525cee0723b012aa016f33cffaf2b391eb8771cecc4f66411a3c869acb74c9" exitCode=0 Oct 30 05:39:36.374319 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:36.374263 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" event={"ID":"74f140c6-30e4-4153-ae27-15d82a63021e","Type":"ContainerDied","Data":"f7525cee0723b012aa016f33cffaf2b391eb8771cecc4f66411a3c869acb74c9"} Oct 30 05:39:37.380996 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.380915 6171 generic.go:356] "Generic (PLEG): container finished" podID="74f140c6-30e4-4153-ae27-15d82a63021e" containerID="c57a2924e07cc52c1a8a770c33b8eea94921d20a0bce90dda40fdfec30f33146" exitCode=0 Oct 30 05:39:37.381464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.380993 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" event={"ID":"74f140c6-30e4-4153-ae27-15d82a63021e","Type":"ContainerDied","Data":"c57a2924e07cc52c1a8a770c33b8eea94921d20a0bce90dda40fdfec30f33146"} Oct 30 05:39:37.382808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.382788 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerID="87ef161a939781226341ae78d9cd78a08171b2b2512ca1388337a596414bb9eb" exitCode=0 Oct 30 05:39:37.382906 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.382871 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" event={"ID":"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8","Type":"ContainerDied","Data":"87ef161a939781226341ae78d9cd78a08171b2b2512ca1388337a596414bb9eb"} Oct 30 05:39:37.514657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.514634 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:37.620205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.620179 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\") pod \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " Oct 30 05:39:37.620302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.620259 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\") pod \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " Oct 30 05:39:37.620302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.620291 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\" (UniqueName: \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\") pod \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\" (UID: \"6a3b6b7c-0a10-455c-a51c-b18368e1008c\") " Oct 30 05:39:37.620674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.620646 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle" pod "6a3b6b7c-0a10-455c-a51c-b18368e1008c" (UID: "6a3b6b7c-0a10-455c-a51c-b18368e1008c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:37.622324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.622303 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457" pod "6a3b6b7c-0a10-455c-a51c-b18368e1008c" (UID: "6a3b6b7c-0a10-455c-a51c-b18368e1008c"). InnerVolumeSpecName "kube-api-access-4g457". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:37.624944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.624922 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util" pod "6a3b6b7c-0a10-455c-a51c-b18368e1008c" (UID: "6a3b6b7c-0a10-455c-a51c-b18368e1008c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:37.721938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.721915 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:37.721938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.721936 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6a3b6b7c-0a10-455c-a51c-b18368e1008c-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:37.722081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:37.721946 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4g457\" (UniqueName: \"kubernetes.io/projected/6a3b6b7c-0a10-455c-a51c-b18368e1008c-kube-api-access-4g457\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.389899 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.389818 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" Oct 30 05:39:38.389899 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.389853 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c629cwv" event={"ID":"6a3b6b7c-0a10-455c-a51c-b18368e1008c","Type":"ContainerDied","Data":"b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2"} Oct 30 05:39:38.389899 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.389897 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b96338be33fa5c5f7b65c3da600e92224d580963959413415935a05e0d381ed2" Oct 30 05:39:38.392245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.392215 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerID="b8e914b567f6adcec7bea1d9c21f9c2317b301b7706177d49662cb8173398d31" exitCode=0 Oct 30 05:39:38.392381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.392307 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" event={"ID":"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8","Type":"ContainerDied","Data":"b8e914b567f6adcec7bea1d9c21f9c2317b301b7706177d49662cb8173398d31"} Oct 30 05:39:38.514020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.514000 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:38.627716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.627685 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\" (UniqueName: \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\") pod \"74f140c6-30e4-4153-ae27-15d82a63021e\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " Oct 30 05:39:38.627853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.627777 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\") pod \"74f140c6-30e4-4153-ae27-15d82a63021e\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " Oct 30 05:39:38.627853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.627839 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\") pod \"74f140c6-30e4-4153-ae27-15d82a63021e\" (UID: \"74f140c6-30e4-4153-ae27-15d82a63021e\") " Oct 30 05:39:38.628438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.628414 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle" pod "74f140c6-30e4-4153-ae27-15d82a63021e" (UID: "74f140c6-30e4-4153-ae27-15d82a63021e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:38.629981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.629955 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k" pod "74f140c6-30e4-4153-ae27-15d82a63021e" (UID: "74f140c6-30e4-4153-ae27-15d82a63021e"). InnerVolumeSpecName "kube-api-access-5qj2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:38.632945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.632924 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util" pod "74f140c6-30e4-4153-ae27-15d82a63021e" (UID: "74f140c6-30e4-4153-ae27-15d82a63021e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:38.729157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.729136 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.729157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.729156 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/74f140c6-30e4-4153-ae27-15d82a63021e-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:38.729270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:38.729165 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qj2k\" (UniqueName: \"kubernetes.io/projected/74f140c6-30e4-4153-ae27-15d82a63021e-kube-api-access-5qj2k\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:39.398134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.398110 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" Oct 30 05:39:39.398134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.398112 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847ah8c52" event={"ID":"74f140c6-30e4-4153-ae27-15d82a63021e","Type":"ContainerDied","Data":"0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7"} Oct 30 05:39:39.398561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.398163 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b9b490481ad8408e4dda91676859e1cd3a03c58b90b9cd1b05d7a7fd6b8fde7" Oct 30 05:39:39.524986 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.524961 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:39.636395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.636374 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\") pod \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " Oct 30 05:39:39.636502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.636404 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\" (UniqueName: \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\") pod \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " Oct 30 05:39:39.636502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.636448 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\") pod \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\" (UID: \"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8\") " Oct 30 05:39:39.636908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.636883 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle" pod "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" (UID: "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:39.638576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.638555 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf" pod "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" (UID: "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8"). InnerVolumeSpecName "kube-api-access-48ssf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:39.641392 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.641357 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util" pod "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" (UID: "5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:39.737923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.737891 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:39.737923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.737915 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:39.737923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:39.737925 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48ssf\" (UniqueName: \"kubernetes.io/projected/5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8-kube-api-access-48ssf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:40.409721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:40.409689 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" Oct 30 05:39:40.409721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:40.409700 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d75wmz5" event={"ID":"5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8","Type":"ContainerDied","Data":"8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9"} Oct 30 05:39:40.410130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:40.409761 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b97f059be5748503612c4d6a3f82d336fab6f0ca964412c6e74d2960c8764a9" Oct 30 05:39:41.415734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:41.415662 6171 generic.go:356] "Generic (PLEG): container finished" podID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerID="ab4ee0280e4e07b92eb106d9157c60c1aad0ff55189384f74233f8f88a84c3d2" exitCode=0 Oct 30 05:39:41.416102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:41.415748 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" event={"ID":"f28dd250-e472-4d5c-b1f9-d747d40edca0","Type":"ContainerDied","Data":"ab4ee0280e4e07b92eb106d9157c60c1aad0ff55189384f74233f8f88a84c3d2"} Oct 30 05:39:42.422683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:42.422651 6171 generic.go:356] "Generic (PLEG): container finished" podID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerID="658bf06ba59b4067c5f2394d0dd2c6fb0899bf191ca4748ca916a87cc69eba11" exitCode=0 Oct 30 05:39:42.423102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:42.422690 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" event={"ID":"f28dd250-e472-4d5c-b1f9-d747d40edca0","Type":"ContainerDied","Data":"658bf06ba59b4067c5f2394d0dd2c6fb0899bf191ca4748ca916a87cc69eba11"} Oct 30 05:39:43.551992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.551971 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:43.669831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.669798 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\") pod \"f28dd250-e472-4d5c-b1f9-d747d40edca0\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " Oct 30 05:39:43.669831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.669834 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\") pod \"f28dd250-e472-4d5c-b1f9-d747d40edca0\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " Oct 30 05:39:43.670009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.669893 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\" (UniqueName: \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\") pod \"f28dd250-e472-4d5c-b1f9-d747d40edca0\" (UID: \"f28dd250-e472-4d5c-b1f9-d747d40edca0\") " Oct 30 05:39:43.670425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.670395 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle" pod "f28dd250-e472-4d5c-b1f9-d747d40edca0" (UID: "f28dd250-e472-4d5c-b1f9-d747d40edca0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:43.672176 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.672152 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc" pod "f28dd250-e472-4d5c-b1f9-d747d40edca0" (UID: "f28dd250-e472-4d5c-b1f9-d747d40edca0"). InnerVolumeSpecName "kube-api-access-jngcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:39:43.674255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.674216 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util" pod "f28dd250-e472-4d5c-b1f9-d747d40edca0" (UID: "f28dd250-e472-4d5c-b1f9-d747d40edca0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:39:43.770979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.770956 6171 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-util\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:43.770979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.770978 6171 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f28dd250-e472-4d5c-b1f9-d747d40edca0-bundle\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:43.771146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:43.770988 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jngcc\" (UniqueName: \"kubernetes.io/projected/f28dd250-e472-4d5c-b1f9-d747d40edca0-kube-api-access-jngcc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:39:44.432960 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:44.432922 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" event={"ID":"f28dd250-e472-4d5c-b1f9-d747d40edca0","Type":"ContainerDied","Data":"00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34"} Oct 30 05:39:44.433122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:44.432977 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv52tf" Oct 30 05:39:44.433122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:39:44.432978 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00a524ff363a60d6209ff1287c706e0c8b2facc2c9f274af1168e9ee9e8ead34" Oct 30 05:39:56.379057 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:39:56.379003 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:40:00.654288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654260 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf"] Oct 30 05:40:00.654640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654608 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="util" Oct 30 05:40:00.654640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654622 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="util" Oct 30 05:40:00.654640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654637 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654642 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654652 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654657 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654664 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654669 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654681 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654685 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654694 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654701 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654709 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654714 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654723 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="util" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654727 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="util" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654735 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="util" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654739 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="util" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654745 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654748 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="pull" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654761 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="extract" Oct 30 05:40:00.654764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654765 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="extract" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654777 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="util" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654784 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="util" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654831 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28dd250-e472-4d5c-b1f9-d747d40edca0" containerName="extract" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654838 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a3b6b7c-0a10-455c-a51c-b18368e1008c" containerName="extract" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654843 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f140c6-30e4-4153-ae27-15d82a63021e" containerName="extract" Oct 30 05:40:00.655326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.654851 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f11bf76-b67a-49cf-84dd-b5a43dd0b9d8" containerName="extract" Oct 30 05:40:00.657993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.657977 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:00.698697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.698664 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8zg\" (UniqueName: \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\") pod \"limitador-operator-controller-manager-856dd584c5-rhxlf\" (UID: \"e149d72e-8e0b-48c6-8d5b-b4b66ea56279\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:00.699291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.699267 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 05:40:00.699459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.699301 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 05:40:00.699459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.699343 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-wt9gc\"" Oct 30 05:40:00.702032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.702007 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf"] Oct 30 05:40:00.799927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.799903 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8zg\" (UniqueName: \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\") pod \"limitador-operator-controller-manager-856dd584c5-rhxlf\" (UID: \"e149d72e-8e0b-48c6-8d5b-b4b66ea56279\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:00.822941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.822919 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8zg\" (UniqueName: \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\") pod \"limitador-operator-controller-manager-856dd584c5-rhxlf\" (UID: \"e149d72e-8e0b-48c6-8d5b-b4b66ea56279\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:00.969826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:00.969773 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:01.269598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:01.269573 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf"] Oct 30 05:40:01.271560 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:40:01.271536 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode149d72e_8e0b_48c6_8d5b_b4b66ea56279.slice/crio-9eca5790fa1de5692c5c2eec9187800736c3b9c224fab09a94520d9f3aade7cb WatchSource:0}: Error finding container 9eca5790fa1de5692c5c2eec9187800736c3b9c224fab09a94520d9f3aade7cb: Status 404 returned error can't find the container with id 9eca5790fa1de5692c5c2eec9187800736c3b9c224fab09a94520d9f3aade7cb Oct 30 05:40:01.510309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:01.510240 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" event={"ID":"e149d72e-8e0b-48c6-8d5b-b4b66ea56279","Type":"ContainerStarted","Data":"9eca5790fa1de5692c5c2eec9187800736c3b9c224fab09a94520d9f3aade7cb"} Oct 30 05:40:04.529323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:04.529278 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" event={"ID":"e149d72e-8e0b-48c6-8d5b-b4b66ea56279","Type":"ContainerStarted","Data":"fcece593e3a5bdff77db5f252ec537f1b18bfbd7707c57a95881927b811eb296"} Oct 30 05:40:04.530250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:04.530226 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:04.581642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:04.581594 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" podStartSLOduration=1.9302849960000001 podStartE2EDuration="4.581582935s" totalImagesPullingTime="2.651297939s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:04.581150729 +0000 UTC m=+463.021527920" watchObservedRunningTime="2026-10-30 05:40:04.581582935 +0000 UTC m=+463.021960127" Oct 30 05:40:16.540677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:16.540648 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:26.378975 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:26.378938 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:40:30.456863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.456829 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf"] Oct 30 05:40:30.457305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.457099 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" containerName="manager" containerID="cri-o://fcece593e3a5bdff77db5f252ec537f1b18bfbd7707c57a95881927b811eb296" gracePeriod=2 Oct 30 05:40:30.468749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.468722 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf"] Oct 30 05:40:30.470428 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:30.470391 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rhxlf\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:30.488476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.488452 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg"] Oct 30 05:40:30.488757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.488744 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" containerName="manager" Oct 30 05:40:30.488757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.488756 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" containerName="manager" Oct 30 05:40:30.488944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.488833 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" containerName="manager" Oct 30 05:40:30.492578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.492557 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:30.508012 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:30.507984 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rhxlf\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:30.522648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.522625 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg"] Oct 30 05:40:30.639608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.639579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fvm8\" (UniqueName: \"kubernetes.io/projected/cc918c8e-0360-430e-a745-2dd3588e0328-kube-api-access-9fvm8\") pod \"limitador-operator-controller-manager-856dd584c5-f8dcg\" (UID: \"cc918c8e-0360-430e-a745-2dd3588e0328\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:30.649865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.649835 6171 generic.go:356] "Generic (PLEG): container finished" podID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" containerID="fcece593e3a5bdff77db5f252ec537f1b18bfbd7707c57a95881927b811eb296" exitCode=0 Oct 30 05:40:30.707714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.707667 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:30.721102 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:30.721077 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rhxlf\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:30.740521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.740498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fvm8\" (UniqueName: \"kubernetes.io/projected/cc918c8e-0360-430e-a745-2dd3588e0328-kube-api-access-9fvm8\") pod \"limitador-operator-controller-manager-856dd584c5-f8dcg\" (UID: \"cc918c8e-0360-430e-a745-2dd3588e0328\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:30.763443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.763420 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fvm8\" (UniqueName: \"kubernetes.io/projected/cc918c8e-0360-430e-a745-2dd3588e0328-kube-api-access-9fvm8\") pod \"limitador-operator-controller-manager-856dd584c5-f8dcg\" (UID: \"cc918c8e-0360-430e-a745-2dd3588e0328\") " pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:30.841594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.841559 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\" (UniqueName: \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\") pod \"e149d72e-8e0b-48c6-8d5b-b4b66ea56279\" (UID: \"e149d72e-8e0b-48c6-8d5b-b4b66ea56279\") " Oct 30 05:40:30.843646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.843614 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg" pod "e149d72e-8e0b-48c6-8d5b-b4b66ea56279" (UID: "e149d72e-8e0b-48c6-8d5b-b4b66ea56279"). InnerVolumeSpecName "kube-api-access-bq8zg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:30.848760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.848743 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:30.942812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:30.942784 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8zg\" (UniqueName: \"kubernetes.io/projected/e149d72e-8e0b-48c6-8d5b-b4b66ea56279-kube-api-access-bq8zg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:40:31.152413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.152393 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg"] Oct 30 05:40:31.154461 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:40:31.154437 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc918c8e_0360_430e_a745_2dd3588e0328.slice/crio-8a9aa31b9d70ed4eb91c50d4a2c8de8fe71cabae192f81a917ac3349b79c5585 WatchSource:0}: Error finding container 8a9aa31b9d70ed4eb91c50d4a2c8de8fe71cabae192f81a917ac3349b79c5585: Status 404 returned error can't find the container with id 8a9aa31b9d70ed4eb91c50d4a2c8de8fe71cabae192f81a917ac3349b79c5585 Oct 30 05:40:31.655723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.655696 6171 scope.go:122] "RemoveContainer" containerID="fcece593e3a5bdff77db5f252ec537f1b18bfbd7707c57a95881927b811eb296" Oct 30 05:40:31.656181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.655703 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:31.657693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.657662 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" event={"ID":"cc918c8e-0360-430e-a745-2dd3588e0328","Type":"ContainerStarted","Data":"26d7adb4e5a3e61101ef83ae091a0c8cc3185398c04cd8bcd3e4f331abc85524"} Oct 30 05:40:31.657796 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.657725 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:31.657796 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.657751 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" event={"ID":"cc918c8e-0360-430e-a745-2dd3588e0328","Type":"ContainerStarted","Data":"8a9aa31b9d70ed4eb91c50d4a2c8de8fe71cabae192f81a917ac3349b79c5585"} Oct 30 05:40:31.668118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.668098 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" path="/var/lib/kubelet/pods/e149d72e-8e0b-48c6-8d5b-b4b66ea56279/volumes" Oct 30 05:40:31.669503 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:31.669485 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-856dd584c5-rhxlf\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e149d72e-8e0b-48c6-8d5b-b4b66ea56279" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-rhxlf" Oct 30 05:40:31.712262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:31.712218 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" podStartSLOduration=1.712202389 podStartE2EDuration="1.712202389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:31.711622391 +0000 UTC m=+490.151999582" watchObservedRunningTime="2026-10-30 05:40:31.712202389 +0000 UTC m=+490.152579581" Oct 30 05:40:42.669067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:40:42.669020 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-856dd584c5-f8dcg" Oct 30 05:40:56.378813 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:40:56.378780 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:41:00.290940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.290904 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 05:41:00.294226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.294204 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.320031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.320007 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 05:41:00.356447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.356422 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-ntk7q\"" Oct 30 05:41:00.360807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.360782 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.360918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.360818 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.360918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.360838 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmw68\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.360918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.360902 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.361099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.360974 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.361099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.361080 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.361195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.361125 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.361195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.361152 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.361261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.361239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.461993 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmw68\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462025 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462062 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462097 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462121 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462139 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462200 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462260 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462293 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462420 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462533 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462796 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462889 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.462971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.462891 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.464348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.464329 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.464703 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.464689 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.499130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.499103 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmw68\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.501385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.501362 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-96dmr\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.606464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.606396 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:00.934670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:00.934650 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 05:41:00.937454 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:41:00.937429 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21930d30_b311_42e1_bdeb_a82bd367255b.slice/crio-7881b0a21147e72e429048ec4058489234f994347fd2259e08d443693716650a WatchSource:0}: Error finding container 7881b0a21147e72e429048ec4058489234f994347fd2259e08d443693716650a: Status 404 returned error can't find the container with id 7881b0a21147e72e429048ec4058489234f994347fd2259e08d443693716650a Oct 30 05:41:01.792860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:01.792824 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerStarted","Data":"7881b0a21147e72e429048ec4058489234f994347fd2259e08d443693716650a"} Oct 30 05:41:03.589018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:03.588988 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:41:03.589305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:03.589070 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:41:03.589305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:03.589098 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:41:03.804783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:03.804750 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerStarted","Data":"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab"} Oct 30 05:41:03.863725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:03.863615 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podStartSLOduration=1.2146036119999999 podStartE2EDuration="3.863593831s" totalImagesPullingTime="2.648990219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:03.861775538 +0000 UTC m=+522.302152731" watchObservedRunningTime="2026-10-30 05:41:03.863593831 +0000 UTC m=+522.303971023" Oct 30 05:41:04.606615 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:04.606581 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:04.611438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:04.611416 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:04.814262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:04.812198 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:04.815807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:04.815784 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:41:13.036356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.036314 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:13.040156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.040134 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.055162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.055145 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 05:41:13.066890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.066871 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-8fkww\"" Oct 30 05:41:13.068415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.068397 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:13.070139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.070122 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.070205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.070148 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4976x\" (UniqueName: \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.167405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.167378 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:13.171150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.171126 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.171239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.171158 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4976x\" (UniqueName: \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.171745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.171727 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.194618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.194591 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4976x\" (UniqueName: \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\") pod \"limitador-limitador-74dfcfb894-fq9hq\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.353408 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.353332 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:13.656488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.656465 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:13.658562 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:41:13.658534 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d6f6d8f_e691_44b5_a0a4_c175ca81bd73.slice/crio-96694c73cbf70a15fcf7c884daee2a07f115b78d50fe0849eb8914e22d7e86a0 WatchSource:0}: Error finding container 96694c73cbf70a15fcf7c884daee2a07f115b78d50fe0849eb8914e22d7e86a0: Status 404 returned error can't find the container with id 96694c73cbf70a15fcf7c884daee2a07f115b78d50fe0849eb8914e22d7e86a0 Oct 30 05:41:13.853012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:13.852985 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" event={"ID":"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73","Type":"ContainerStarted","Data":"96694c73cbf70a15fcf7c884daee2a07f115b78d50fe0849eb8914e22d7e86a0"} Oct 30 05:41:14.419830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.419793 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:41:14.423429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.423408 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.444619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.444594 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:41:14.482698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.482678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3509408-6100-43da-8ff8-fd3b5768bcaa-config-file\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.482815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.482716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5bj\" (UniqueName: \"kubernetes.io/projected/d3509408-6100-43da-8ff8-fd3b5768bcaa-kube-api-access-mx5bj\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.488915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.488888 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:41:14.583344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.583315 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3509408-6100-43da-8ff8-fd3b5768bcaa-config-file\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.583492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.583383 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5bj\" (UniqueName: \"kubernetes.io/projected/d3509408-6100-43da-8ff8-fd3b5768bcaa-kube-api-access-mx5bj\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.583949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.583927 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/d3509408-6100-43da-8ff8-fd3b5768bcaa-config-file\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.607246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.607219 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5bj\" (UniqueName: \"kubernetes.io/projected/d3509408-6100-43da-8ff8-fd3b5768bcaa-kube-api-access-mx5bj\") pod \"limitador-limitador-7c9b87d574-h82bs\" (UID: \"d3509408-6100-43da-8ff8-fd3b5768bcaa\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.739860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.739835 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:14.858613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.858579 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" event={"ID":"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73","Type":"ContainerStarted","Data":"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d"} Oct 30 05:41:14.858754 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.858688 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:14.900570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:14.900526 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" podStartSLOduration=0.838293811 podStartE2EDuration="1.90051453s" totalImagesPullingTime="1.062220719s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:14.898490689 +0000 UTC m=+533.338867882" watchObservedRunningTime="2026-10-30 05:41:14.90051453 +0000 UTC m=+533.340891722" Oct 30 05:41:15.043181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:15.043158 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:41:15.045092 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:41:15.045038 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3509408_6100_43da_8ff8_fd3b5768bcaa.slice/crio-ad3beb26685243c9b6ccb6a383cc5320659b95de65479d633e00a50efa87b023 WatchSource:0}: Error finding container ad3beb26685243c9b6ccb6a383cc5320659b95de65479d633e00a50efa87b023: Status 404 returned error can't find the container with id ad3beb26685243c9b6ccb6a383cc5320659b95de65479d633e00a50efa87b023 Oct 30 05:41:15.864625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:15.864585 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" event={"ID":"d3509408-6100-43da-8ff8-fd3b5768bcaa","Type":"ContainerStarted","Data":"a8cb88aff2625467a27d428f8c45df49fa765885ed956a99819d8b3502ffe678"} Oct 30 05:41:15.866854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:15.864650 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" event={"ID":"d3509408-6100-43da-8ff8-fd3b5768bcaa","Type":"ContainerStarted","Data":"ad3beb26685243c9b6ccb6a383cc5320659b95de65479d633e00a50efa87b023"} Oct 30 05:41:15.866854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:15.864704 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:15.908890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:15.908849 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" podStartSLOduration=1.9088350630000002 podStartE2EDuration="1.908835063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:41:15.908086214 +0000 UTC m=+534.348463408" watchObservedRunningTime="2026-10-30 05:41:15.908835063 +0000 UTC m=+534.349212258" Oct 30 05:41:25.865784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:25.865745 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:26.379120 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:41:26.379084 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:41:26.871533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:26.871505 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-h82bs" Oct 30 05:41:26.940454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:26.940422 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:26.940618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:26.940599 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" podUID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" containerName="limitador" containerID="cri-o://fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d" gracePeriod=30 Oct 30 05:41:27.881753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.881729 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:27.924818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.924787 6171 generic.go:356] "Generic (PLEG): container finished" podID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" containerID="fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d" exitCode=0 Oct 30 05:41:27.924985 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.924830 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" Oct 30 05:41:27.924985 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.924848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" event={"ID":"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73","Type":"ContainerDied","Data":"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d"} Oct 30 05:41:27.924985 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.924902 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-fq9hq" event={"ID":"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73","Type":"ContainerDied","Data":"96694c73cbf70a15fcf7c884daee2a07f115b78d50fe0849eb8914e22d7e86a0"} Oct 30 05:41:27.924985 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.924926 6171 scope.go:122] "RemoveContainer" containerID="fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d" Oct 30 05:41:27.933945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.933923 6171 scope.go:122] "RemoveContainer" containerID="fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d" Oct 30 05:41:27.934348 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:41:27.934322 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d\": container with ID starting with fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d not found: ID does not exist" containerID="fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d" Oct 30 05:41:27.934417 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.934363 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d"} err="failed to get container status \"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d\": rpc error: code = NotFound desc = could not find container \"fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d\": container with ID starting with fbe69661734c7e27b68e56a40a3e613ad42b1ed50d693c9d1d05360da789378d not found: ID does not exist" Oct 30 05:41:28.000001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:27.999964 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\" (UniqueName: \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\") pod \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " Oct 30 05:41:28.000220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.000072 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\" (UniqueName: \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\") pod \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\" (UID: \"2d6f6d8f-e691-44b5-a0a4-c175ca81bd73\") " Oct 30 05:41:28.000413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.000389 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file" pod "2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" (UID: "2d6f6d8f-e691-44b5-a0a4-c175ca81bd73"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:41:28.002279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.002250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x" pod "2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" (UID: "2d6f6d8f-e691-44b5-a0a4-c175ca81bd73"). InnerVolumeSpecName "kube-api-access-4976x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:41:28.101030 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.100983 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4976x\" (UniqueName: \"kubernetes.io/projected/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-kube-api-access-4976x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:41:28.101030 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.101020 6171 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73-config-file\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:41:28.270983 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.270890 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:28.283124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:28.283094 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-fq9hq"] Oct 30 05:41:29.669921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:41:29.669888 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" path="/var/lib/kubelet/pods/2d6f6d8f-e691-44b5-a0a4-c175ca81bd73/volumes" Oct 30 05:41:56.379359 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:41:56.379322 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:42:26.379412 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:42:26.379377 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:42:35.285339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:35.285308 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:42:45.292554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.292521 6171 generic.go:356] "Generic (PLEG): container finished" podID="21930d30-b311-42e1-bdeb-a82bd367255b" containerID="764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab" exitCode=0 Oct 30 05:42:45.292987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.292595 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerDied","Data":"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab"} Oct 30 05:42:45.292987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.292963 6171 scope.go:122] "RemoveContainer" containerID="764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab" Oct 30 05:42:45.294205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.294171 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:42:45.294273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.294249 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:42:45.294310 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:45.294278 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:42:46.310308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:46.310271 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerStarted","Data":"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4"} Oct 30 05:42:46.607541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:46.607443 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:42:46.608779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:46.608752 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:42:46.608917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:46.608811 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 05:42:47.606943 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:47.606906 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:42:47.607382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:47.606969 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 05:42:48.607757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:48.607714 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:42:48.608212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:48.607785 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 05:42:49.607147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:49.607105 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:42:49.607449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:49.607418 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 05:42:49.812933 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:49.812894 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:42:50.607617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:50.607570 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:42:50.607839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:50.607631 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" Oct 30 05:42:51.610681 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:51.610647 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:42:51.611860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:42:51.611839 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 05:42:56.380748 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:42:56.380711 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:43:26.380002 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:43:26.379962 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:43:34.471287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.471249 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t"] Oct 30 05:43:34.471886 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.471648 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" containerName="limitador" Oct 30 05:43:34.471886 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.471661 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" containerName="limitador" Oct 30 05:43:34.471886 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.471726 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d6f6d8f-e691-44b5-a0a4-c175ca81bd73" containerName="limitador" Oct 30 05:43:34.474774 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.474755 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.490779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.490749 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 05:43:34.493732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.493702 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 05:43:34.505994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.505968 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-2k6w7\"" Oct 30 05:43:34.506190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.506103 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 05:43:34.507573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.507548 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t"] Oct 30 05:43:34.521270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521237 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthqg\" (UniqueName: \"kubernetes.io/projected/9c5b4302-435f-4bc0-8578-73fa12bdb419-kube-api-access-sthqg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.521429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521284 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5b4302-435f-4bc0-8578-73fa12bdb419-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.521429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521374 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.521429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521415 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.521613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521505 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.521613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.521546 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622215 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthqg\" (UniqueName: \"kubernetes.io/projected/9c5b4302-435f-4bc0-8578-73fa12bdb419-kube-api-access-sthqg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622282 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5b4302-435f-4bc0-8578-73fa12bdb419-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622330 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622401 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622766 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622773 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.622848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.622820 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.624804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.624781 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9c5b4302-435f-4bc0-8578-73fa12bdb419-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.624961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.624943 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9c5b4302-435f-4bc0-8578-73fa12bdb419-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.646298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.646266 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthqg\" (UniqueName: \"kubernetes.io/projected/9c5b4302-435f-4bc0-8578-73fa12bdb419-kube-api-access-sthqg\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-7gh9t\" (UID: \"9c5b4302-435f-4bc0-8578-73fa12bdb419\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:34.787709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:34.787666 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:35.131741 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:35.131712 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t"] Oct 30 05:43:35.134428 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:43:35.134394 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c5b4302_435f_4bc0_8578_73fa12bdb419.slice/crio-1dda907e18c2ced9f29cc65e579680d2995ebf81385bf1a2ae2267d7c658246b WatchSource:0}: Error finding container 1dda907e18c2ced9f29cc65e579680d2995ebf81385bf1a2ae2267d7c658246b: Status 404 returned error can't find the container with id 1dda907e18c2ced9f29cc65e579680d2995ebf81385bf1a2ae2267d7c658246b Oct 30 05:43:35.141494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:35.136101 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:43:35.547173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:35.547128 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" event={"ID":"9c5b4302-435f-4bc0-8578-73fa12bdb419","Type":"ContainerStarted","Data":"1dda907e18c2ced9f29cc65e579680d2995ebf81385bf1a2ae2267d7c658246b"} Oct 30 05:43:35.675013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:35.674983 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:43:36.554104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.553982 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" event={"ID":"9c5b4302-435f-4bc0-8578-73fa12bdb419","Type":"ContainerStarted","Data":"f800de9ce98a40410101aaf58d08ba7de1321e0a9f38b94719c0dfb3c5ff3b37"} Oct 30 05:43:36.590853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.590808 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp"] Oct 30 05:43:36.594504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.594479 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.599648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.599603 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" podStartSLOduration=1.476331557 podStartE2EDuration="2.599589121s" totalImagesPullingTime="1.123257564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:36.598099367 +0000 UTC m=+675.038476563" watchObservedRunningTime="2026-10-30 05:43:36.599589121 +0000 UTC m=+675.039966357" Oct 30 05:43:36.612861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.612833 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 05:43:36.624733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.624702 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp"] Oct 30 05:43:36.637451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637420 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.637618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637463 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.637618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637552 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e622128-5588-4764-91dd-f7759834b7eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.637618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637602 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.637738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637619 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.637738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.637662 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vnf7\" (UniqueName: \"kubernetes.io/projected/3e622128-5588-4764-91dd-f7759834b7eb-kube-api-access-7vnf7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738505 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vnf7\" (UniqueName: \"kubernetes.io/projected/3e622128-5588-4764-91dd-f7759834b7eb-kube-api-access-7vnf7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738570 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738618 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738645 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e622128-5588-4764-91dd-f7759834b7eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738692 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.738759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.738712 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.739026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.739000 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.739026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.739033 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.739241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.739196 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.741054 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.741018 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e622128-5588-4764-91dd-f7759834b7eb-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.741400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.741383 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e622128-5588-4764-91dd-f7759834b7eb-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.762438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.762411 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vnf7\" (UniqueName: \"kubernetes.io/projected/3e622128-5588-4764-91dd-f7759834b7eb-kube-api-access-7vnf7\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-q74rp\" (UID: \"3e622128-5588-4764-91dd-f7759834b7eb\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:36.787836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.787804 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:36.907783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:36.907701 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:37.218448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:37.218423 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp"] Oct 30 05:43:37.220295 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:43:37.220272 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e622128_5588_4764_91dd_f7759834b7eb.slice/crio-87165b7afa8df1eb514f3a43d19d66dc2cc8dd1cb22a688d0848aeabf4d9b5b3 WatchSource:0}: Error finding container 87165b7afa8df1eb514f3a43d19d66dc2cc8dd1cb22a688d0848aeabf4d9b5b3: Status 404 returned error can't find the container with id 87165b7afa8df1eb514f3a43d19d66dc2cc8dd1cb22a688d0848aeabf4d9b5b3 Oct 30 05:43:37.560590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:37.560550 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" event={"ID":"3e622128-5588-4764-91dd-f7759834b7eb","Type":"ContainerStarted","Data":"af95548d25c4512b57762c417d3db22de3456d43db94a72e97ac183c1c200dc8"} Oct 30 05:43:37.561101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:37.560606 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" event={"ID":"3e622128-5588-4764-91dd-f7759834b7eb","Type":"ContainerStarted","Data":"87165b7afa8df1eb514f3a43d19d66dc2cc8dd1cb22a688d0848aeabf4d9b5b3"} Oct 30 05:43:37.604256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:37.604202 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" podStartSLOduration=1.395441503 podStartE2EDuration="1.604186034s" totalImagesPullingTime="208.744531ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:37.603240338 +0000 UTC m=+676.043617532" watchObservedRunningTime="2026-10-30 05:43:37.604186034 +0000 UTC m=+676.044563227" Oct 30 05:43:37.908266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:37.908236 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:39.487361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:39.487330 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:43:43.908110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:43.908071 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:43:44.498285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.498250 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t"] Oct 30 05:43:44.502520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.502500 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.530751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.530727 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 05:43:44.532759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.532732 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t"] Oct 30 05:43:44.608397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608356 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.608612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608415 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.608612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608585 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653bfbea-50ba-4751-b13e-da9ef57544bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.608747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608642 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtwns\" (UniqueName: \"kubernetes.io/projected/653bfbea-50ba-4751-b13e-da9ef57544bd-kube-api-access-mtwns\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.608747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608691 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.608747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.608721 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710027 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653bfbea-50ba-4751-b13e-da9ef57544bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710101 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtwns\" (UniqueName: \"kubernetes.io/projected/653bfbea-50ba-4751-b13e-da9ef57544bd-kube-api-access-mtwns\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710183 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710214 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710288 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710335 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.710980 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.710772 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.711130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.711117 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.711213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.711134 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.713067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.713017 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/653bfbea-50ba-4751-b13e-da9ef57544bd-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.713706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.713676 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/653bfbea-50ba-4751-b13e-da9ef57544bd-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.734614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.734577 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtwns\" (UniqueName: \"kubernetes.io/projected/653bfbea-50ba-4751-b13e-da9ef57544bd-kube-api-access-mtwns\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t\" (UID: \"653bfbea-50ba-4751-b13e-da9ef57544bd\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:44.788609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.788529 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:44.807997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.807971 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:44.818317 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:44.818295 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:45.126394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.126370 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t"] Oct 30 05:43:45.129555 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:43:45.129521 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653bfbea_50ba_4751_b13e_da9ef57544bd.slice/crio-da1b7526a025f9ad97d054d32b9a4523d522e5ecdd866175d3b5916b6d8720fc WatchSource:0}: Error finding container da1b7526a025f9ad97d054d32b9a4523d522e5ecdd866175d3b5916b6d8720fc: Status 404 returned error can't find the container with id da1b7526a025f9ad97d054d32b9a4523d522e5ecdd866175d3b5916b6d8720fc Oct 30 05:43:45.601614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.601579 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" event={"ID":"653bfbea-50ba-4751-b13e-da9ef57544bd","Type":"ContainerStarted","Data":"9365ada5669eb52d0766e2e31b907fbff93746464a097a3f066d2b48bf93a79d"} Oct 30 05:43:45.601794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.601635 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" event={"ID":"653bfbea-50ba-4751-b13e-da9ef57544bd","Type":"ContainerStarted","Data":"da1b7526a025f9ad97d054d32b9a4523d522e5ecdd866175d3b5916b6d8720fc"} Oct 30 05:43:45.612685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.612660 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" Oct 30 05:43:45.644235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.644151 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" podStartSLOduration=1.4366767839999999 podStartE2EDuration="1.644138348s" totalImagesPullingTime="207.461564ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:45.642892322 +0000 UTC m=+684.083269515" watchObservedRunningTime="2026-10-30 05:43:45.644138348 +0000 UTC m=+684.084515537" Oct 30 05:43:45.819068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:45.819018 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:46.908628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:46.908597 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:46.921560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:46.921540 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:47.635402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:47.635376 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" Oct 30 05:43:48.456060 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:48.456018 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:43:49.946510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:49.946462 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg"] Oct 30 05:43:49.950290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:49.950266 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:49.970123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:49.970100 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg"] Oct 30 05:43:49.977454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:49.977435 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 30 05:43:50.056906 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.056879 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.057101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.056911 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.057101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.056931 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.057101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.057001 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2bp9\" (UniqueName: \"kubernetes.io/projected/6c89f1f8-53fb-4c78-b610-f0709cd448a5-kube-api-access-c2bp9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.057101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.057083 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.057239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.057113 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158134 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2bp9\" (UniqueName: \"kubernetes.io/projected/6c89f1f8-53fb-4c78-b610-f0709cd448a5-kube-api-access-c2bp9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158188 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158356 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158443 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158475 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158506 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158785 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.158809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158802 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.159131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.158842 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.160567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.160540 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c89f1f8-53fb-4c78-b610-f0709cd448a5-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.160869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.160853 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c89f1f8-53fb-4c78-b610-f0709cd448a5-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.181640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.181615 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2bp9\" (UniqueName: \"kubernetes.io/projected/6c89f1f8-53fb-4c78-b610-f0709cd448a5-kube-api-access-c2bp9\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-crblg\" (UID: \"6c89f1f8-53fb-4c78-b610-f0709cd448a5\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.266654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.266620 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:50.580807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.580765 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg"] Oct 30 05:43:50.582122 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:43:50.582097 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c89f1f8_53fb_4c78_b610_f0709cd448a5.slice/crio-9e615616f9927fe0b4547ab6d67033b77089fb23d999dd579be9e9219e9cdc11 WatchSource:0}: Error finding container 9e615616f9927fe0b4547ab6d67033b77089fb23d999dd579be9e9219e9cdc11: Status 404 returned error can't find the container with id 9e615616f9927fe0b4547ab6d67033b77089fb23d999dd579be9e9219e9cdc11 Oct 30 05:43:50.636724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:50.636699 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" event={"ID":"6c89f1f8-53fb-4c78-b610-f0709cd448a5","Type":"ContainerStarted","Data":"9e615616f9927fe0b4547ab6d67033b77089fb23d999dd579be9e9219e9cdc11"} Oct 30 05:43:51.644066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:51.644016 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" event={"ID":"6c89f1f8-53fb-4c78-b610-f0709cd448a5","Type":"ContainerStarted","Data":"a6775a1aa94d2463132e449ce70dd41f1debc3f75047d6f5afa1966784ace598"} Oct 30 05:43:51.690363 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:51.690312 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" podStartSLOduration=2.497570332 podStartE2EDuration="2.690297212s" totalImagesPullingTime="192.72688ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:51.689297429 +0000 UTC m=+690.129674621" watchObservedRunningTime="2026-10-30 05:43:51.690297212 +0000 UTC m=+690.130674405" Oct 30 05:43:52.267552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:52.267511 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:43:54.819208 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:54.819175 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:54.831474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:54.831446 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:55.628724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:55.628692 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:43:55.676632 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:43:55.676607 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" Oct 30 05:43:56.380745 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:43:56.380710 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:44:00.266916 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:00.266874 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:44:00.279656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:00.279621 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:44:00.702592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:00.702508 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-crblg" Oct 30 05:44:08.485402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.485359 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:08.489516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.489491 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.519265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.519236 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-xfkjz\"" Oct 30 05:44:08.519265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.519236 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 05:44:08.519469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.519404 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 05:44:08.522716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.522680 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:08.639868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.639834 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.640094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.639874 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqf8l\" (UniqueName: \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.740567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.740479 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.740567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.740516 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqf8l\" (UniqueName: \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.743147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.743128 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.767879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.767845 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqf8l\" (UniqueName: \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\") pod \"ai-gateway-controller-84dff4756b-zmrk4\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:08.803783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:08.803747 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:09.110599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:09.110572 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:09.112751 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:44:09.112724 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod443de169_c8ab_4127_afd7_aec7684cac0e.slice/crio-cb5e27ab73cfdeb4384ce711e39c748d27a40ba42d3191639c61070c911fbacc WatchSource:0}: Error finding container cb5e27ab73cfdeb4384ce711e39c748d27a40ba42d3191639c61070c911fbacc: Status 404 returned error can't find the container with id cb5e27ab73cfdeb4384ce711e39c748d27a40ba42d3191639c61070c911fbacc Oct 30 05:44:09.739134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:09.739087 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" event={"ID":"443de169-c8ab-4127-afd7-aec7684cac0e","Type":"ContainerStarted","Data":"cb5e27ab73cfdeb4384ce711e39c748d27a40ba42d3191639c61070c911fbacc"} Oct 30 05:44:11.750975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:11.750938 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" event={"ID":"443de169-c8ab-4127-afd7-aec7684cac0e","Type":"ContainerStarted","Data":"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e"} Oct 30 05:44:11.751468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:11.751070 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:11.802717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:11.802662 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" podStartSLOduration=1.577159054 podStartE2EDuration="3.802648564s" totalImagesPullingTime="2.22548951s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:11.800655787 +0000 UTC m=+710.241032982" watchObservedRunningTime="2026-10-30 05:44:11.802648564 +0000 UTC m=+710.243025756" Oct 30 05:44:13.514821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.514782 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-bq9hh"] Oct 30 05:44:13.519032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.519008 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.539203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.539172 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-bq9hh"] Oct 30 05:44:13.630714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.630685 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 05:44:13.684425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.684377 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5911c2-912e-496c-b4f9-13fe71875626-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.684597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.684492 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkq2h\" (UniqueName: \"kubernetes.io/projected/7e5911c2-912e-496c-b4f9-13fe71875626-kube-api-access-bkq2h\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.785194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.785101 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5911c2-912e-496c-b4f9-13fe71875626-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.785194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.785164 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkq2h\" (UniqueName: \"kubernetes.io/projected/7e5911c2-912e-496c-b4f9-13fe71875626-kube-api-access-bkq2h\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.785813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.785791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5911c2-912e-496c-b4f9-13fe71875626-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.808682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.808653 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkq2h\" (UniqueName: \"kubernetes.io/projected/7e5911c2-912e-496c-b4f9-13fe71875626-kube-api-access-bkq2h\") pod \"payload-processing-55fbbbf9dc-bq9hh\" (UID: \"7e5911c2-912e-496c-b4f9-13fe71875626\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:13.832580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:13.832549 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:14.138780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:14.138670 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-bq9hh"] Oct 30 05:44:14.141593 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:44:14.141561 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5911c2_912e_496c_b4f9_13fe71875626.slice/crio-002121081e15d02a2cf6312d055ae43a67e775cdb32bde77c5037292cc7fa114 WatchSource:0}: Error finding container 002121081e15d02a2cf6312d055ae43a67e775cdb32bde77c5037292cc7fa114: Status 404 returned error can't find the container with id 002121081e15d02a2cf6312d055ae43a67e775cdb32bde77c5037292cc7fa114 Oct 30 05:44:14.769596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:14.769554 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" event={"ID":"7e5911c2-912e-496c-b4f9-13fe71875626","Type":"ContainerStarted","Data":"002121081e15d02a2cf6312d055ae43a67e775cdb32bde77c5037292cc7fa114"} Oct 30 05:44:16.411504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.411420 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds"] Oct 30 05:44:16.415211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.415192 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.452230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.452194 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds"] Oct 30 05:44:16.611553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611516 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxwgz\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-kube-api-access-kxwgz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611574 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611656 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611703 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611817 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611890 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611921 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.611988 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.611966 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.612117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.612101 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713643 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxwgz\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-kube-api-access-kxwgz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713697 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713737 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713766 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713797 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713864 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713886 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.713975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.713959 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.714372 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.714207 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.714654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.714536 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.714654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.714606 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.714790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.714681 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.714896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.714874 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.716815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.716793 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.716915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.716822 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.737542 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.737505 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.737667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.737589 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxwgz\" (UniqueName: \"kubernetes.io/projected/b78e85c7-fea8-49e0-af5a-802c1ebe3ea9-kube-api-access-kxwgz\") pod \"maas-default-gateway-openshift-default-58847f9fc5-rxgds\" (UID: \"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:16.782613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.782571 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" event={"ID":"7e5911c2-912e-496c-b4f9-13fe71875626","Type":"ContainerStarted","Data":"705948a5c9dc0a39629d53451a3fe050e8435af205d139f229b840833616da06"} Oct 30 05:44:16.782789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.782655 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:16.829637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:16.829588 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" podStartSLOduration=1.974649974 podStartE2EDuration="3.829573218s" totalImagesPullingTime="1.854923244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:16.828147234 +0000 UTC m=+715.268524426" watchObservedRunningTime="2026-10-30 05:44:16.829573218 +0000 UTC m=+715.269950410" Oct 30 05:44:17.027737 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.027702 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:17.364138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.364112 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds"] Oct 30 05:44:17.366191 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:44:17.366156 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb78e85c7_fea8_49e0_af5a_802c1ebe3ea9.slice/crio-8ca4f3b9df15bd93855b32d99e48ef0933678574b4619bf912ddbf1027c3812a WatchSource:0}: Error finding container 8ca4f3b9df15bd93855b32d99e48ef0933678574b4619bf912ddbf1027c3812a: Status 404 returned error can't find the container with id 8ca4f3b9df15bd93855b32d99e48ef0933678574b4619bf912ddbf1027c3812a Oct 30 05:44:17.368579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.368545 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:17.368687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.368626 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:17.368687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.368675 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:17.793814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.793032 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" event={"ID":"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9","Type":"ContainerStarted","Data":"571b51e7664fd25c0052075f1748443d1398ddcf9cb9dc38f39f2a61c829c339"} Oct 30 05:44:17.793814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.793121 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" event={"ID":"b78e85c7-fea8-49e0-af5a-802c1ebe3ea9","Type":"ContainerStarted","Data":"8ca4f3b9df15bd93855b32d99e48ef0933678574b4619bf912ddbf1027c3812a"} Oct 30 05:44:17.848794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:17.848732 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podStartSLOduration=1.8487145470000002 podStartE2EDuration="1.848714547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:17.84774404 +0000 UTC m=+716.288121234" watchObservedRunningTime="2026-10-30 05:44:17.848714547 +0000 UTC m=+716.289091743" Oct 30 05:44:18.027943 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:18.027900 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:18.029390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:18.029366 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:18.029538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:18.029419 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:19.028445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:19.028407 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:19.028865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:19.028461 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:20.028637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:20.028596 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:20.029089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:20.028656 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:21.028138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:21.028099 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:21.028327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:21.028157 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:22.029149 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:22.029106 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:22.029763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:22.029177 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:22.758390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:22.758359 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:23.028452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.028360 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:23.028452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.028429 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:23.075981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.075941 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-654df4bd7-95hcs"] Oct 30 05:44:23.081178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.081151 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.106070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.101449 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-654df4bd7-95hcs"] Oct 30 05:44:23.167426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.167392 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e57dbc4-5ff8-4158-981f-a3b456468cb3-metrics-certs\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.167599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.167543 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv5wj\" (UniqueName: \"kubernetes.io/projected/6e57dbc4-5ff8-4158-981f-a3b456468cb3-kube-api-access-jv5wj\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.268992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.268945 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv5wj\" (UniqueName: \"kubernetes.io/projected/6e57dbc4-5ff8-4158-981f-a3b456468cb3-kube-api-access-jv5wj\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.269235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.269030 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e57dbc4-5ff8-4158-981f-a3b456468cb3-metrics-certs\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.271654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.271624 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e57dbc4-5ff8-4158-981f-a3b456468cb3-metrics-certs\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.305845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.305762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv5wj\" (UniqueName: \"kubernetes.io/projected/6e57dbc4-5ff8-4158-981f-a3b456468cb3-kube-api-access-jv5wj\") pod \"ai-gateway-controller-654df4bd7-95hcs\" (UID: \"6e57dbc4-5ff8-4158-981f-a3b456468cb3\") " pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.418778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.418736 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:23.727430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.727407 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-654df4bd7-95hcs"] Oct 30 05:44:23.729644 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:44:23.729617 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e57dbc4_5ff8_4158_981f_a3b456468cb3.slice/crio-434f06917035b62034178d8d91baf27d560bea23aeac936b76d50971c53737cc WatchSource:0}: Error finding container 434f06917035b62034178d8d91baf27d560bea23aeac936b76d50971c53737cc: Status 404 returned error can't find the container with id 434f06917035b62034178d8d91baf27d560bea23aeac936b76d50971c53737cc Oct 30 05:44:23.836775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:23.836737 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" event={"ID":"6e57dbc4-5ff8-4158-981f-a3b456468cb3","Type":"ContainerStarted","Data":"434f06917035b62034178d8d91baf27d560bea23aeac936b76d50971c53737cc"} Oct 30 05:44:24.028845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:24.028756 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:24.028845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:24.028814 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:24.844214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:24.844177 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" event={"ID":"6e57dbc4-5ff8-4158-981f-a3b456468cb3","Type":"ContainerStarted","Data":"9f05b38bd0ce01c8355d71444e1dde698bf1eea67f27d57947226749e963a4d4"} Oct 30 05:44:24.844694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:24.844267 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:24.887253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:24.887202 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" podStartSLOduration=1.493732821 podStartE2EDuration="1.887186336s" totalImagesPullingTime="393.453515ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:24.885517744 +0000 UTC m=+723.325894940" watchObservedRunningTime="2026-10-30 05:44:24.887186336 +0000 UTC m=+723.327563529" Oct 30 05:44:25.028806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:25.028766 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" start-of-body= Oct 30 05:44:25.028959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:25.028839 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": dial tcp 10.133.0.42:15021: connect: connection refused" Oct 30 05:44:26.274248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:26.274218 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:26.274758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:26.274600 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:26.275388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:26.275368 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" Oct 30 05:44:26.380246 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:44:26.380209 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:44:27.796574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:27.796546 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-bq9hh" Oct 30 05:44:31.471761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.471679 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7"] Oct 30 05:44:31.475443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.475421 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.518133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.518088 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7"] Oct 30 05:44:31.544081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544024 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544085 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544163 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544196 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544330 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544370 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tblv\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-kube-api-access-9tblv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544446 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544467 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cabe1b-3080-4d86-b821-a1e469c7439e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.544532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.544505 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.645787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.645746 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.645800 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tblv\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-kube-api-access-9tblv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.645842 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.645870 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cabe1b-3080-4d86-b821-a1e469c7439e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646019 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646075 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646102 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646132 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646358 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646435 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646702 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646682 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.646979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.646957 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b3cabe1b-3080-4d86-b821-a1e469c7439e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.648452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.648425 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.648687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.648672 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.669526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.669497 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tblv\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-kube-api-access-9tblv\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.670266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.670240 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b3cabe1b-3080-4d86-b821-a1e469c7439e-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-84wh7\" (UID: \"b3cabe1b-3080-4d86-b821-a1e469c7439e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:31.790172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:31.790083 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:32.113826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.113794 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7"] Oct 30 05:44:32.114655 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:44:32.114618 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3cabe1b_3080_4d86_b821_a1e469c7439e.slice/crio-4012858cf7226dcb7b5577a3b8c6bad7c553487378799344d2920c05b32014c7 WatchSource:0}: Error finding container 4012858cf7226dcb7b5577a3b8c6bad7c553487378799344d2920c05b32014c7: Status 404 returned error can't find the container with id 4012858cf7226dcb7b5577a3b8c6bad7c553487378799344d2920c05b32014c7 Oct 30 05:44:32.116847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.116810 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.116929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.116881 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.116929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.116911 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:44:32.911341 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.911301 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" event={"ID":"b3cabe1b-3080-4d86-b821-a1e469c7439e","Type":"ContainerStarted","Data":"a6c785ef2a66d0ae0b5a41941325b5abd1ca56ff3c9352d5cfb96675bce690fc"} Oct 30 05:44:32.911783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.911359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" event={"ID":"b3cabe1b-3080-4d86-b821-a1e469c7439e","Type":"ContainerStarted","Data":"4012858cf7226dcb7b5577a3b8c6bad7c553487378799344d2920c05b32014c7"} Oct 30 05:44:32.969119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:32.969066 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podStartSLOduration=1.9690352880000002 podStartE2EDuration="1.969035288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:32.968509412 +0000 UTC m=+731.408886608" watchObservedRunningTime="2026-10-30 05:44:32.969035288 +0000 UTC m=+731.409412480" Oct 30 05:44:33.796588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:33.796494 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:33.802126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:33.801755 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:33.802126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:33.801813 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:34.795517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:34.795479 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:34.796179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:34.796141 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:35.805476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:35.805435 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:35.805889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:35.805496 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:35.851807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:35.851777 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-654df4bd7-95hcs" Oct 30 05:44:35.921905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:35.921868 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:35.922177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:35.922132 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" podUID="443de169-c8ab-4127-afd7-aec7684cac0e" containerName="manager" containerID="cri-o://63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e" gracePeriod=30 Oct 30 05:44:36.213307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.213271 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:36.312787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.312756 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\") pod \"443de169-c8ab-4127-afd7-aec7684cac0e\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " Oct 30 05:44:36.312953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.312924 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\" (UniqueName: \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\") pod \"443de169-c8ab-4127-afd7-aec7684cac0e\" (UID: \"443de169-c8ab-4127-afd7-aec7684cac0e\") " Oct 30 05:44:36.315184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.315149 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs" pod "443de169-c8ab-4127-afd7-aec7684cac0e" (UID: "443de169-c8ab-4127-afd7-aec7684cac0e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:44:36.315312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.315208 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l" pod "443de169-c8ab-4127-afd7-aec7684cac0e" (UID: "443de169-c8ab-4127-afd7-aec7684cac0e"). InnerVolumeSpecName "kube-api-access-sqf8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:44:36.414790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.414698 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqf8l\" (UniqueName: \"kubernetes.io/projected/443de169-c8ab-4127-afd7-aec7684cac0e-kube-api-access-sqf8l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:44:36.414790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.414736 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/443de169-c8ab-4127-afd7-aec7684cac0e-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:44:36.799952 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.799488 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:36.799952 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.799555 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:36.935459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.935427 6171 generic.go:356] "Generic (PLEG): container finished" podID="443de169-c8ab-4127-afd7-aec7684cac0e" containerID="63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e" exitCode=0 Oct 30 05:44:36.935967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.935488 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" event={"ID":"443de169-c8ab-4127-afd7-aec7684cac0e","Type":"ContainerDied","Data":"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e"} Oct 30 05:44:36.935967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.935528 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" event={"ID":"443de169-c8ab-4127-afd7-aec7684cac0e","Type":"ContainerDied","Data":"cb5e27ab73cfdeb4384ce711e39c748d27a40ba42d3191639c61070c911fbacc"} Oct 30 05:44:36.935967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.935552 6171 scope.go:122] "RemoveContainer" containerID="63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e" Oct 30 05:44:36.935967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.935487 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-84dff4756b-zmrk4" Oct 30 05:44:36.944918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.944897 6171 scope.go:122] "RemoveContainer" containerID="63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e" Oct 30 05:44:36.945243 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:44:36.945225 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e\": container with ID starting with 63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e not found: ID does not exist" containerID="63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e" Oct 30 05:44:36.945300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:36.945254 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e"} err="failed to get container status \"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e\": rpc error: code = NotFound desc = could not find container \"63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e\": container with ID starting with 63f2c900a6e378438b8d1c64454af0452a9441bd08658244ff6b5f364560ec0e not found: ID does not exist" Oct 30 05:44:37.005395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:37.005354 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:37.007187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:37.007158 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-84dff4756b-zmrk4"] Oct 30 05:44:37.670198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:37.670163 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="443de169-c8ab-4127-afd7-aec7684cac0e" path="/var/lib/kubelet/pods/443de169-c8ab-4127-afd7-aec7684cac0e/volumes" Oct 30 05:44:37.801073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:37.797123 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:37.801073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:37.797195 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:38.791138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:38.791100 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:38.791539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:38.791163 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:39.790823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:39.790787 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" start-of-body= Oct 30 05:44:39.791012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:39.790847 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": dial tcp 10.133.0.44:15021: connect: connection refused" Oct 30 05:44:41.075132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:41.075099 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:41.075611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:41.075471 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:41.076289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:44:41.076270 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" Oct 30 05:44:56.386576 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:44:56.386538 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:45:00.159697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.159657 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:00.160324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.160307 6171 cpu_manager.go:414] "containerMap: removing container" podUID="443de169-c8ab-4127-afd7-aec7684cac0e" containerName="manager" Oct 30 05:45:00.160377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.160326 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="443de169-c8ab-4127-afd7-aec7684cac0e" containerName="manager" Oct 30 05:45:00.160478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.160466 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="443de169-c8ab-4127-afd7-aec7684cac0e" containerName="manager" Oct 30 05:45:00.169788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.169763 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.183336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.183306 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:00.192671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.192640 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:45:00.192857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.192732 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:45:00.192857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.192736 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:45:00.250674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.250632 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq86r\" (UniqueName: \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.251053 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.251021 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.352025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.351986 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq86r\" (UniqueName: \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.352228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.352075 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.352876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.352851 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.375808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.375771 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq86r\" (UniqueName: \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\") pod \"maas-api-key-cleanup-29888985-gvwzf\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.484727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.484695 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:00.788292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:00.788269 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:00.790247 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:00.790221 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabbdec48_c0b0_4e11_b870_88f7ef54538b.slice/crio-bc34b93a5b7cc8017bdaac007542cfbbb58c407624bd7b29026bd306921e8eed WatchSource:0}: Error finding container bc34b93a5b7cc8017bdaac007542cfbbb58c407624bd7b29026bd306921e8eed: Status 404 returned error can't find the container with id bc34b93a5b7cc8017bdaac007542cfbbb58c407624bd7b29026bd306921e8eed Oct 30 05:45:01.064721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:01.064639 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerStarted","Data":"bc34b93a5b7cc8017bdaac007542cfbbb58c407624bd7b29026bd306921e8eed"} Oct 30 05:45:04.083617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.083584 6171 generic.go:356] "Generic (PLEG): container finished" podID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerID="74e463eea03c5bbfd636be027d1eb91da85b8a023e1604c414fdbde064ac4d8c" exitCode=60 Oct 30 05:45:04.084020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.083616 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerDied","Data":"74e463eea03c5bbfd636be027d1eb91da85b8a023e1604c414fdbde064ac4d8c"} Oct 30 05:45:04.084020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.083957 6171 scope.go:122] "RemoveContainer" containerID="74e463eea03c5bbfd636be027d1eb91da85b8a023e1604c414fdbde064ac4d8c" Oct 30 05:45:04.484691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.484657 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 05:45:04.488738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.488713 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.505155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.505128 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4a68a1-openshift-default-dockercfg-mbxx7\"" Oct 30 05:45:04.510836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.510801 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 05:45:04.595239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595194 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595316 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595381 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595414 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595445 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595478 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfmwm\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595501 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595607 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595564 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.595607 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.595593 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696639 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696529 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfmwm\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696639 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696601 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696639 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696632 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696656 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696714 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696784 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696854 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.696926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696912 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.696938 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.697140 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.697190 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.697492 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.697570 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.697738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.697713 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.699564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.699542 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.699903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.699886 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.725615 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.725587 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfmwm\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.725885 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.725868 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:04.802628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:04.802595 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:05.065105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.065069 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 05:45:05.068668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.068649 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.092340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.092310 6171 generic.go:356] "Generic (PLEG): container finished" podID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerID="b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608" exitCode=60 Oct 30 05:45:05.092761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.092387 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerDied","Data":"b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608"} Oct 30 05:45:05.092761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.092460 6171 scope.go:122] "RemoveContainer" containerID="74e463eea03c5bbfd636be027d1eb91da85b8a023e1604c414fdbde064ac4d8c" Oct 30 05:45:05.092761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.092754 6171 scope.go:122] "RemoveContainer" containerID="b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608" Oct 30 05:45:05.093156 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:45:05.093081 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gvwzf_odh-ai-gateway-infra(abbdec48-c0b0-4e11-b870-88f7ef54538b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" Oct 30 05:45:05.103508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.103482 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 05:45:05.111908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.111884 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.111934 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112013 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112157 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nd2zh\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112280 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112324 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112366 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112390 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.112438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.112424 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.118954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.118938 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-e2e3d0-openshift-default-dockercfg-dkznd\"" Oct 30 05:45:05.131670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.131644 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 05:45:05.134898 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:05.134874 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb434846d_a8ea_4eec_a759_e5bd74ca12e8.slice/crio-9ae96323026ba9cda980470b76c0104e7ee731d2f5b0b8a169bedd62841d23aa WatchSource:0}: Error finding container 9ae96323026ba9cda980470b76c0104e7ee731d2f5b0b8a169bedd62841d23aa: Status 404 returned error can't find the container with id 9ae96323026ba9cda980470b76c0104e7ee731d2f5b0b8a169bedd62841d23aa Oct 30 05:45:05.138034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.137993 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.138209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.138110 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.138209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.138156 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.213527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213494 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213545 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213589 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213637 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213691 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213761 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213788 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.213905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213846 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nd2zh\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.214147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.213950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.214788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.214765 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.215136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.215112 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.215381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.215360 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.215492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.215469 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.217559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.217531 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.218016 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.217992 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.225569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.225541 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:45:05.236625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.236566 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.245020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.244989 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nd2zh\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.246910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.246830 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.257766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.257741 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-a35835-openshift-default-dockercfg-rbcbq\"" Oct 30 05:45:05.258014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.257988 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:45:05.315296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315254 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315367 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315407 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315533 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcmmp\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315589 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315777 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.315879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.315863 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.383230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.383196 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:05.416526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416489 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.416720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416541 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.416720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416599 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcmmp\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.416919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416765 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.416919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416842 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.416915 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.417000 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.417105 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.417165 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.417278 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.417752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.417730 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.418032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.418011 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.418332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.418186 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.418332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.418301 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.419837 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.419776 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.420749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.420715 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.441131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.441096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.441131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.441127 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcmmp\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.554644 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.554547 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:05.729428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.729400 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 05:45:05.730870 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:05.730843 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75b417bf_5723_485e_bedc_c57bfe9465b9.slice/crio-f8e114d0d22a690fa772fde5068fd45802cf09c98cab12f32144e7d8a314b27a WatchSource:0}: Error finding container f8e114d0d22a690fa772fde5068fd45802cf09c98cab12f32144e7d8a314b27a: Status 404 returned error can't find the container with id f8e114d0d22a690fa772fde5068fd45802cf09c98cab12f32144e7d8a314b27a Oct 30 05:45:05.733231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.733204 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.733291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.733262 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.733328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.733291 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.871380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.871353 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:45:05.873623 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:05.873590 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec17335d_d8f7_40f6_a3ef_646b49be1384.slice/crio-ad72806739a87945facfdee3113d152a21f551c07b11733effb5e88be6e52411 WatchSource:0}: Error finding container ad72806739a87945facfdee3113d152a21f551c07b11733effb5e88be6e52411: Status 404 returned error can't find the container with id ad72806739a87945facfdee3113d152a21f551c07b11733effb5e88be6e52411 Oct 30 05:45:05.876081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.876030 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.876170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.876156 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:05.876226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:05.876196 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:06.102131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.102011 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerStarted","Data":"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5"} Oct 30 05:45:06.102131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.102111 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerStarted","Data":"f8e114d0d22a690fa772fde5068fd45802cf09c98cab12f32144e7d8a314b27a"} Oct 30 05:45:06.105188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.105163 6171 scope.go:122] "RemoveContainer" containerID="b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608" Oct 30 05:45:06.105571 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:45:06.105430 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gvwzf_odh-ai-gateway-infra(abbdec48-c0b0-4e11-b870-88f7ef54538b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" Oct 30 05:45:06.106876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.106848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerStarted","Data":"223228bd9680e59c6a9eb9831156a259b57da898246a1bad4823232487811608"} Oct 30 05:45:06.106957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.106889 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerStarted","Data":"ad72806739a87945facfdee3113d152a21f551c07b11733effb5e88be6e52411"} Oct 30 05:45:06.109225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.109203 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerStarted","Data":"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742"} Oct 30 05:45:06.109307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.109249 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerStarted","Data":"9ae96323026ba9cda980470b76c0104e7ee731d2f5b0b8a169bedd62841d23aa"} Oct 30 05:45:06.155343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.155164 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podStartSLOduration=1.155126403 podStartE2EDuration="1.155126403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.147594965 +0000 UTC m=+764.587972173" watchObservedRunningTime="2026-10-30 05:45:06.155126403 +0000 UTC m=+764.595503594" Oct 30 05:45:06.194932 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.194862 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podStartSLOduration=1.194842704 podStartE2EDuration="1.194842704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.192368947 +0000 UTC m=+764.632746151" watchObservedRunningTime="2026-10-30 05:45:06.194842704 +0000 UTC m=+764.635219898" Oct 30 05:45:06.280076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.279990 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podStartSLOduration=2.279975487 podStartE2EDuration="2.279975487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.277715134 +0000 UTC m=+764.718092328" watchObservedRunningTime="2026-10-30 05:45:06.279975487 +0000 UTC m=+764.720352739" Oct 30 05:45:06.409507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.409418 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:06.417579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.417537 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:06.555218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.555184 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:06.560678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.560653 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:06.803847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.803811 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:06.809031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:06.809008 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:07.115630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.115541 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:07.115630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.115592 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:07.115630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.115616 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:07.116291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.116274 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:07.116384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.116339 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:07.116440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:07.116426 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:10.361302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.361263 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 05:45:10.365202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.365181 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.388163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.388143 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-c78066-dockercfg-4stn8\"" Oct 30 05:45:10.391093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.391067 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 05:45:10.400341 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.400318 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-c78066\"" Oct 30 05:45:10.402837 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.402814 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.403006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.402991 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snkrl\" (UniqueName: \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.446289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.446265 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 05:45:10.450112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.450094 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.481119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.481096 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 05:45:10.503921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.503898 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fpvl\" (UniqueName: \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.504055 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.503962 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.504055 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.504023 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snkrl\" (UniqueName: \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.504147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.504095 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.504794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.504777 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.529338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.529310 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snkrl\" (UniqueName: \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\") pod \"payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.606609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.606550 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fpvl\" (UniqueName: \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.606815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.606618 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.607536 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.607498 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.636214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.635951 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fpvl\" (UniqueName: \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\") pod \"payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.678656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.678624 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:10.762390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.762359 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:10.984808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:10.984784 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 05:45:10.986646 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:10.986623 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881a8306_55c5_47bf_a93b_dbced02b54bc.slice/crio-9877fed7340c7f02cc8e888f874a731e152253c262575ec430d852acdc20e44c WatchSource:0}: Error finding container 9877fed7340c7f02cc8e888f874a731e152253c262575ec430d852acdc20e44c: Status 404 returned error can't find the container with id 9877fed7340c7f02cc8e888f874a731e152253c262575ec430d852acdc20e44c Oct 30 05:45:11.074180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.074153 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 05:45:11.075185 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:11.075162 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdfe32e8_d06f_4f3f_8cff_d7ddd1b6fcf0.slice/crio-4da1e5bceee0d5221ba313c4970a0f532f576ad53f153211ea3e9f9f0977abe9 WatchSource:0}: Error finding container 4da1e5bceee0d5221ba313c4970a0f532f576ad53f153211ea3e9f9f0977abe9: Status 404 returned error can't find the container with id 4da1e5bceee0d5221ba313c4970a0f532f576ad53f153211ea3e9f9f0977abe9 Oct 30 05:45:11.140102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.140073 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" event={"ID":"881a8306-55c5-47bf-a93b-dbced02b54bc","Type":"ContainerStarted","Data":"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1"} Oct 30 05:45:11.140253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.140134 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" event={"ID":"881a8306-55c5-47bf-a93b-dbced02b54bc","Type":"ContainerStarted","Data":"9877fed7340c7f02cc8e888f874a731e152253c262575ec430d852acdc20e44c"} Oct 30 05:45:11.140321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.140276 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:11.141887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.141864 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" event={"ID":"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0","Type":"ContainerStarted","Data":"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f"} Oct 30 05:45:11.141993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.141906 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" event={"ID":"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0","Type":"ContainerStarted","Data":"4da1e5bceee0d5221ba313c4970a0f532f576ad53f153211ea3e9f9f0977abe9"} Oct 30 05:45:11.141993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.141966 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:11.182327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.182277 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" podStartSLOduration=1.182262573 podStartE2EDuration="1.182262573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:11.180086648 +0000 UTC m=+769.620463844" watchObservedRunningTime="2026-10-30 05:45:11.182262573 +0000 UTC m=+769.622639782" Oct 30 05:45:11.220288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.220239 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" podStartSLOduration=1.220225886 podStartE2EDuration="1.220225886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:11.219531405 +0000 UTC m=+769.659908597" watchObservedRunningTime="2026-10-30 05:45:11.220225886 +0000 UTC m=+769.660603078" Oct 30 05:45:11.849301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.849264 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 05:45:11.853255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.853232 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:11.862333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.862302 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 05:45:11.871908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.871879 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-197e9b\"" Oct 30 05:45:11.872970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.872943 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:11.873686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.873660 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 05:45:11.880783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.880764 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-197e9b-dockercfg-h2c8z\"" Oct 30 05:45:11.887012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.886990 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 05:45:11.919477 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.919449 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzx6z\" (UniqueName: \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:11.919637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.919494 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:11.919637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.919601 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:11.919713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:11.919641 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw4mh\" (UniqueName: \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.024950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.024901 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.025155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.024990 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw4mh\" (UniqueName: \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.025155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.025087 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzx6z\" (UniqueName: \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:12.025155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.025137 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:12.025838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.025797 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.026684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.026663 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:12.048929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.048897 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzx6z\" (UniqueName: \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\") pod \"payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:12.049095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.048909 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw4mh\" (UniqueName: \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\") pod \"payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.173679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.173581 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:12.190935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.190139 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:12.506904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.506876 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 05:45:12.509297 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:12.509253 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3517a55f_318e_4335_a017_dd112e066158.slice/crio-34e1f3444b7f0754d60b765b4c2674aa5dbbca30eccecd271b6d1be15a84803c WatchSource:0}: Error finding container 34e1f3444b7f0754d60b765b4c2674aa5dbbca30eccecd271b6d1be15a84803c: Status 404 returned error can't find the container with id 34e1f3444b7f0754d60b765b4c2674aa5dbbca30eccecd271b6d1be15a84803c Oct 30 05:45:12.549808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.549763 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 05:45:12.552510 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:12.552477 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8d300b1_06e7_42f5_afbe_0bdeb7957a9e.slice/crio-ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8 WatchSource:0}: Error finding container ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8: Status 404 returned error can't find the container with id ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8 Oct 30 05:45:12.630096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.630057 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 05:45:12.634199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.634168 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.651584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.651547 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-a35835\"" Oct 30 05:45:12.654788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.654755 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 05:45:12.667817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.667778 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 05:45:12.679674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.679642 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.697012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.696966 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 05:45:12.741959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.741915 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.742416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.742380 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw2kq\" (UniqueName: \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.742597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.742572 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.742680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.742662 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnnvd\" (UniqueName: \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.743193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.743098 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-a35835-dockercfg-sdz78\"" Oct 30 05:45:12.844134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.844004 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.844134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.844102 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnnvd\" (UniqueName: \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.844384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.844161 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.844384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.844250 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw2kq\" (UniqueName: \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.844825 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.844798 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.845177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.845154 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.869891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.869852 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw2kq\" (UniqueName: \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\") pod \"payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:12.870295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.869971 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnnvd\" (UniqueName: \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\") pod \"payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:12.949358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:12.949323 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:13.001453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.001408 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:13.158216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.158107 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" event={"ID":"3517a55f-318e-4335-a017-dd112e066158","Type":"ContainerStarted","Data":"c37e289861f68f8052ae47d4c60817980b44d9b6c36d3a6953288d2f6ebd33fd"} Oct 30 05:45:13.158216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.158177 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:13.158216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.158198 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" event={"ID":"3517a55f-318e-4335-a017-dd112e066158","Type":"ContainerStarted","Data":"34e1f3444b7f0754d60b765b4c2674aa5dbbca30eccecd271b6d1be15a84803c"} Oct 30 05:45:13.160079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.160013 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" event={"ID":"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e","Type":"ContainerStarted","Data":"d47039c82d3ea0c131acfa52505478ad1418906dbd3d1b92c58e37d127594681"} Oct 30 05:45:13.160079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.160077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" event={"ID":"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e","Type":"ContainerStarted","Data":"ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8"} Oct 30 05:45:13.160307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.160188 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:13.198978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.198906 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" podStartSLOduration=2.198884856 podStartE2EDuration="2.198884856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:13.198229578 +0000 UTC m=+771.638606773" watchObservedRunningTime="2026-10-30 05:45:13.198884856 +0000 UTC m=+771.639262050" Oct 30 05:45:13.241908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.241847 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" podStartSLOduration=2.241832792 podStartE2EDuration="2.241832792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:13.240849692 +0000 UTC m=+771.681226888" watchObservedRunningTime="2026-10-30 05:45:13.241832792 +0000 UTC m=+771.682209984" Oct 30 05:45:13.259382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.259353 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 05:45:13.260298 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:13.260273 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114634d3_0554_4801_8c34_866fc41d1666.slice/crio-b3f45ec5d3817f822bdc36fff77c63df53f40ee2c4fc6975dfbf658d3dd5a7ab WatchSource:0}: Error finding container b3f45ec5d3817f822bdc36fff77c63df53f40ee2c4fc6975dfbf658d3dd5a7ab: Status 404 returned error can't find the container with id b3f45ec5d3817f822bdc36fff77c63df53f40ee2c4fc6975dfbf658d3dd5a7ab Oct 30 05:45:13.323264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.323241 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 05:45:13.324185 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:13.324154 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d5001f6_5cd5_43de_aa4c_d47ddd912639.slice/crio-18239f2cc51f0910b272de5fb98a149197949b9e2bf3a322f3ae83e8e8608913 WatchSource:0}: Error finding container 18239f2cc51f0910b272de5fb98a149197949b9e2bf3a322f3ae83e8e8608913: Status 404 returned error can't find the container with id 18239f2cc51f0910b272de5fb98a149197949b9e2bf3a322f3ae83e8e8608913 Oct 30 05:45:13.907937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:13.907896 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:14.167949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.167840 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" event={"ID":"4d5001f6-5cd5-43de-aa4c-d47ddd912639","Type":"ContainerStarted","Data":"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4"} Oct 30 05:45:14.167949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.167911 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" event={"ID":"4d5001f6-5cd5-43de-aa4c-d47ddd912639","Type":"ContainerStarted","Data":"18239f2cc51f0910b272de5fb98a149197949b9e2bf3a322f3ae83e8e8608913"} Oct 30 05:45:14.167949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.167937 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:14.169947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.169915 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" event={"ID":"114634d3-0554-4801-8c34-866fc41d1666","Type":"ContainerStarted","Data":"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c"} Oct 30 05:45:14.170131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.169974 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" event={"ID":"114634d3-0554-4801-8c34-866fc41d1666","Type":"ContainerStarted","Data":"b3f45ec5d3817f822bdc36fff77c63df53f40ee2c4fc6975dfbf658d3dd5a7ab"} Oct 30 05:45:14.170289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.170273 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:14.231072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.223840 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" podStartSLOduration=2.223818755 podStartE2EDuration="2.223818755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:14.222915494 +0000 UTC m=+772.663292688" watchObservedRunningTime="2026-10-30 05:45:14.223818755 +0000 UTC m=+772.664195950" Oct 30 05:45:14.266720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.266667 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" podStartSLOduration=2.266653135 podStartE2EDuration="2.266653135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:14.265109267 +0000 UTC m=+772.705486460" watchObservedRunningTime="2026-10-30 05:45:14.266653135 +0000 UTC m=+772.707030328" Oct 30 05:45:14.691868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.691827 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 05:45:14.696101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.696081 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.714853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.714821 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-e2e3d0\"" Oct 30 05:45:14.715127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.715106 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-e2e3d0-dockercfg-v9z8c\"" Oct 30 05:45:14.718623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.718597 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 05:45:14.747951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.747912 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 05:45:14.752308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.752282 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.771701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.771673 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 05:45:14.783806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.783773 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9wnn\" (UniqueName: \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.783958 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.783824 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.784018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.783942 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc2v4\" (UniqueName: \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.784018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.783989 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.885458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.885397 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc2v4\" (UniqueName: \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.885458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.885466 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.885746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.885598 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9wnn\" (UniqueName: \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.885746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.885654 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.886413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.886385 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.886413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.886406 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:14.910854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.910814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc2v4\" (UniqueName: \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\") pod \"payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:14.911258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:14.910929 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9wnn\" (UniqueName: \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\") pod \"payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:15.011039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.010998 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:15.067317 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.067281 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:15.344973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.344941 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 05:45:15.347073 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:15.347025 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9689235_35dd_4ef4_a797_8c0e7c123edd.slice/crio-bce56b2866fc43e0b969a3a4e4b04c16e10bff6ca557e72410c7b416fc2c5767 WatchSource:0}: Error finding container bce56b2866fc43e0b969a3a4e4b04c16e10bff6ca557e72410c7b416fc2c5767: Status 404 returned error can't find the container with id bce56b2866fc43e0b969a3a4e4b04c16e10bff6ca557e72410c7b416fc2c5767 Oct 30 05:45:15.402585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.402548 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 05:45:15.403465 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:15.403435 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3ad0c3_ff77_4811_b3be_e55e91ed545d.slice/crio-b0dda93e09b0c46484a1a7dba04ecab2419b805d1a72eeaa83fad0cb4c613e9e WatchSource:0}: Error finding container b0dda93e09b0c46484a1a7dba04ecab2419b805d1a72eeaa83fad0cb4c613e9e: Status 404 returned error can't find the container with id b0dda93e09b0c46484a1a7dba04ecab2419b805d1a72eeaa83fad0cb4c613e9e Oct 30 05:45:15.562200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.562166 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 05:45:15.566163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.566141 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.588869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.588840 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 05:45:15.593320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.593290 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqchk\" (UniqueName: \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.593510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.593352 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.657152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.650484 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4a68a1-dockercfg-pv6td\"" Oct 30 05:45:15.657152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.650602 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4a68a1\"" Oct 30 05:45:15.698377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.698349 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqchk\" (UniqueName: \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.698587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.698487 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.699450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.699422 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.722777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.722750 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqchk\" (UniqueName: \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\") pod \"payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:15.881841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:15.881750 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:16.183978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.183872 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" event={"ID":"5b3ad0c3-ff77-4811-b3be-e55e91ed545d","Type":"ContainerStarted","Data":"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4"} Oct 30 05:45:16.183978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.183945 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:16.183978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.183965 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" event={"ID":"5b3ad0c3-ff77-4811-b3be-e55e91ed545d","Type":"ContainerStarted","Data":"b0dda93e09b0c46484a1a7dba04ecab2419b805d1a72eeaa83fad0cb4c613e9e"} Oct 30 05:45:16.185691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.185665 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" event={"ID":"e9689235-35dd-4ef4-a797-8c0e7c123edd","Type":"ContainerStarted","Data":"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214"} Oct 30 05:45:16.185813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.185704 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" event={"ID":"e9689235-35dd-4ef4-a797-8c0e7c123edd","Type":"ContainerStarted","Data":"bce56b2866fc43e0b969a3a4e4b04c16e10bff6ca557e72410c7b416fc2c5767"} Oct 30 05:45:16.185813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.185738 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:16.229763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.223238 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 05:45:16.235023 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.232992 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" podStartSLOduration=2.23297105 podStartE2EDuration="2.23297105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:16.223715063 +0000 UTC m=+774.664092258" watchObservedRunningTime="2026-10-30 05:45:16.23297105 +0000 UTC m=+774.673348247" Oct 30 05:45:16.266745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.266696 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" podStartSLOduration=2.266680938 podStartE2EDuration="2.266680938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:16.264824601 +0000 UTC m=+774.705201796" watchObservedRunningTime="2026-10-30 05:45:16.266680938 +0000 UTC m=+774.707058131" Oct 30 05:45:16.825255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.825219 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 05:45:16.829385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.829364 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:16.846507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.846477 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-1354a5\"" Oct 30 05:45:16.850099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.850074 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 05:45:16.910762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.910731 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:16.910918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:16.910895 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dcbr\" (UniqueName: \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.013489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.013442 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dcbr\" (UniqueName: \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.013656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.013579 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.014521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.014491 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.045066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.041838 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dcbr\" (UniqueName: \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\") pod \"payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.143552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.143469 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:17.199362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.199321 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" event={"ID":"fd5a63a7-b570-42ba-ba6e-322ece01cea2","Type":"ContainerStarted","Data":"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9"} Oct 30 05:45:17.200006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.199400 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" event={"ID":"fd5a63a7-b570-42ba-ba6e-322ece01cea2","Type":"ContainerStarted","Data":"ad3b4ecf2334d415127e5237133ba917e8d1556d6e4fa234170e6af52a8e309e"} Oct 30 05:45:17.200305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.200283 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:17.250448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.250395 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" podStartSLOduration=2.250382282 podStartE2EDuration="2.250382282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:17.249018975 +0000 UTC m=+775.689396169" watchObservedRunningTime="2026-10-30 05:45:17.250382282 +0000 UTC m=+775.690759541" Oct 30 05:45:17.464874 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.464843 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 05:45:17.466095 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:17.466064 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbfcb39_a7e8_4a91_a196_639a1daa4dd2.slice/crio-5b0db99bea4194c257fff7e9396e42779eede3a334a4f8c7a177021fc18f4138 WatchSource:0}: Error finding container 5b0db99bea4194c257fff7e9396e42779eede3a334a4f8c7a177021fc18f4138: Status 404 returned error can't find the container with id 5b0db99bea4194c257fff7e9396e42779eede3a334a4f8c7a177021fc18f4138 Oct 30 05:45:17.544001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.543971 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 05:45:17.547826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.547806 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.558714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.558689 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 05:45:17.562651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.562631 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.568075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.568019 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-128436-dockercfg-92lst\"" Oct 30 05:45:17.572073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.572029 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 05:45:17.588370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.588329 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 05:45:17.623230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.623198 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzrw\" (UniqueName: \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.623390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.623297 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w85tx\" (UniqueName: \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.623430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.623393 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.623493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.623463 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.626901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.626881 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-128436\"" Oct 30 05:45:17.724161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.724130 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w85tx\" (UniqueName: \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.724352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.724195 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.724352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.724247 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.724352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.724296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzrw\" (UniqueName: \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.725096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.725071 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.725216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.725169 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.747722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.747690 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w85tx\" (UniqueName: \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\") pod \"payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.747846 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.747726 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzrw\" (UniqueName: \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\") pod \"payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:17.862810 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.862770 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:17.879402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:17.879181 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:18.201109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.201069 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 05:45:18.212937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.212908 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" event={"ID":"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2","Type":"ContainerStarted","Data":"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154"} Oct 30 05:45:18.213141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.212962 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" event={"ID":"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2","Type":"ContainerStarted","Data":"5b0db99bea4194c257fff7e9396e42779eede3a334a4f8c7a177021fc18f4138"} Oct 30 05:45:18.213141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.213007 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:18.223174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.223087 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 05:45:18.225676 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:18.225646 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f16ffec_8432_4229_ab73_0c5ea222d7f9.slice/crio-aafb06735403064c8096663bab89c082dba52c3440ea4206d27b7e9b363dd2d1 WatchSource:0}: Error finding container aafb06735403064c8096663bab89c082dba52c3440ea4206d27b7e9b363dd2d1: Status 404 returned error can't find the container with id aafb06735403064c8096663bab89c082dba52c3440ea4206d27b7e9b363dd2d1 Oct 30 05:45:18.530405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.530291 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" podStartSLOduration=2.530273811 podStartE2EDuration="2.530273811s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:18.262994516 +0000 UTC m=+776.703371709" watchObservedRunningTime="2026-10-30 05:45:18.530273811 +0000 UTC m=+776.970651004" Oct 30 05:45:18.532424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.532391 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:18.663855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:18.663819 6171 scope.go:122] "RemoveContainer" containerID="b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608" Oct 30 05:45:19.226857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.226802 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" event={"ID":"010116a0-d561-46eb-b06c-48c6988d1adc","Type":"ContainerStarted","Data":"df7fe6473084770231a637b9241f2c2aa974ec00a7047d43197c2ee6e048aa64"} Oct 30 05:45:19.228700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.227572 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" event={"ID":"010116a0-d561-46eb-b06c-48c6988d1adc","Type":"ContainerStarted","Data":"e7c51c3702376485bf56992aebbad932cd3431bd1f1e7d64391c8e611548a157"} Oct 30 05:45:19.228700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.228638 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:19.233106 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.232474 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerStarted","Data":"1dabdcfc03d98418f96a3315ca62e55cbfa491a03013cb95118022d489c94bdc"} Oct 30 05:45:19.235715 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.235670 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" event={"ID":"9f16ffec-8432-4229-ab73-0c5ea222d7f9","Type":"ContainerStarted","Data":"afc7bffa44bfb0d749b3f309ca38bcfecf7f08381f0f5fb8aa5a9489731f7189"} Oct 30 05:45:19.235872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.235736 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" event={"ID":"9f16ffec-8432-4229-ab73-0c5ea222d7f9","Type":"ContainerStarted","Data":"aafb06735403064c8096663bab89c082dba52c3440ea4206d27b7e9b363dd2d1"} Oct 30 05:45:19.269811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.269755 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" podStartSLOduration=2.269739378 podStartE2EDuration="2.269739378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:19.268244633 +0000 UTC m=+777.708621825" watchObservedRunningTime="2026-10-30 05:45:19.269739378 +0000 UTC m=+777.710116572" Oct 30 05:45:19.313591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.313493 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" podStartSLOduration=2.313473569 podStartE2EDuration="2.313473569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:19.311560075 +0000 UTC m=+777.751937268" watchObservedRunningTime="2026-10-30 05:45:19.313473569 +0000 UTC m=+777.753850764" Oct 30 05:45:19.353369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:19.353307 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" podStartSLOduration=17.347422624 podStartE2EDuration="19.353291103s" totalImagesPullingTime="2.005868479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:19.350929752 +0000 UTC m=+777.791306946" watchObservedRunningTime="2026-10-30 05:45:19.353291103 +0000 UTC m=+777.793668357" Oct 30 05:45:20.244492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.244454 6171 generic.go:356] "Generic (PLEG): container finished" podID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerID="1dabdcfc03d98418f96a3315ca62e55cbfa491a03013cb95118022d489c94bdc" exitCode=60 Oct 30 05:45:20.244951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.244526 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerDied","Data":"1dabdcfc03d98418f96a3315ca62e55cbfa491a03013cb95118022d489c94bdc"} Oct 30 05:45:20.244951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.244586 6171 scope.go:122] "RemoveContainer" containerID="b50d44cc55c35ce45cabc3b875102f1d52680fa7e1176194ae7e01a8434bf608" Oct 30 05:45:20.244951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.244934 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:20.245253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.245220 6171 scope.go:122] "RemoveContainer" containerID="1dabdcfc03d98418f96a3315ca62e55cbfa491a03013cb95118022d489c94bdc" Oct 30 05:45:20.245464 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:45:20.245441 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-gvwzf_odh-ai-gateway-infra(abbdec48-c0b0-4e11-b870-88f7ef54538b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" Oct 30 05:45:20.361717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:20.361682 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:21.458652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.458623 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:21.565133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.565036 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\") pod \"abbdec48-c0b0-4e11-b870-88f7ef54538b\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " Oct 30 05:45:21.565281 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.565172 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\" (UniqueName: \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\") pod \"abbdec48-c0b0-4e11-b870-88f7ef54538b\" (UID: \"abbdec48-c0b0-4e11-b870-88f7ef54538b\") " Oct 30 05:45:21.565382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.565361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca" pod "abbdec48-c0b0-4e11-b870-88f7ef54538b" (UID: "abbdec48-c0b0-4e11-b870-88f7ef54538b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:21.565548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.565534 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/abbdec48-c0b0-4e11-b870-88f7ef54538b-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.567549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.567528 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r" pod "abbdec48-c0b0-4e11-b870-88f7ef54538b" (UID: "abbdec48-c0b0-4e11-b870-88f7ef54538b"). InnerVolumeSpecName "kube-api-access-kq86r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:21.666145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:21.666115 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kq86r\" (UniqueName: \"kubernetes.io/projected/abbdec48-c0b0-4e11-b870-88f7ef54538b-kube-api-access-kq86r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:45:22.151188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.151151 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 05:45:22.151394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.151348 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 05:45:22.261177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.261031 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" event={"ID":"abbdec48-c0b0-4e11-b870-88f7ef54538b","Type":"ContainerDied","Data":"bc34b93a5b7cc8017bdaac007542cfbbb58c407624bd7b29026bd306921e8eed"} Oct 30 05:45:22.261343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.261214 6171 scope.go:122] "RemoveContainer" containerID="1dabdcfc03d98418f96a3315ca62e55cbfa491a03013cb95118022d489c94bdc" Oct 30 05:45:22.261386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.261366 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf" Oct 30 05:45:22.326227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.326193 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:22.332277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:22.332244 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-gvwzf"] Oct 30 05:45:23.612766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:23.612725 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:23.681276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:23.681239 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" path="/var/lib/kubelet/pods/abbdec48-c0b0-4e11-b870-88f7ef54538b/volumes" Oct 30 05:45:24.181653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:24.180595 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 05:45:24.185379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:24.185348 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 05:45:25.178903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:25.178872 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 05:45:25.179344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:25.179329 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 05:45:26.410370 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:45:26.410325 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:45:27.220073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:27.219552 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 05:45:27.226400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:27.225421 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 05:45:29.240424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:29.240377 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 05:45:29.776175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:29.776133 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:30.251680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:30.250953 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 05:45:31.257078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.256997 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 05:45:31.258407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.257440 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 05:45:31.391512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.391471 6171 generic.go:356] "Generic (PLEG): container finished" podID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerID="223228bd9680e59c6a9eb9831156a259b57da898246a1bad4823232487811608" exitCode=0 Oct 30 05:45:31.391700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.391555 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerDied","Data":"223228bd9680e59c6a9eb9831156a259b57da898246a1bad4823232487811608"} Oct 30 05:45:31.392086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.392032 6171 scope.go:122] "RemoveContainer" containerID="223228bd9680e59c6a9eb9831156a259b57da898246a1bad4823232487811608" Oct 30 05:45:31.394285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.394246 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:31.394611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.394589 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:31.394787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:31.394760 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:32.417507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:32.417468 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerStarted","Data":"fc738844dd2fd83640952652ab5a3d8135e40883cdd48b6fa81fc825f38ebacd"} Oct 30 05:45:32.555177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:32.555126 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:32.556758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:32.556725 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:32.556933 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:32.556782 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:33.555029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:33.554983 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:33.555588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:33.555127 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:34.556392 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:34.556350 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:34.556967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:34.556430 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:35.446222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.446186 6171 generic.go:356] "Generic (PLEG): container finished" podID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerID="20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742" exitCode=0 Oct 30 05:45:35.446464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.446246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerDied","Data":"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742"} Oct 30 05:45:35.446678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.446662 6171 scope.go:122] "RemoveContainer" containerID="20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742" Oct 30 05:45:35.448348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.448315 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:35.448455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.448396 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:35.448455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.448446 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:35.555080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.555020 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:35.555252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:35.555099 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:36.456254 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.456202 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerStarted","Data":"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb"} Oct 30 05:45:36.458454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.458422 6171 generic.go:356] "Generic (PLEG): container finished" podID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerID="026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5" exitCode=0 Oct 30 05:45:36.458582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.458511 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerDied","Data":"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5"} Oct 30 05:45:36.458846 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.458832 6171 scope.go:122] "RemoveContainer" containerID="026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5" Oct 30 05:45:36.460471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.460446 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:36.460670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.460645 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:36.460783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.460770 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:45:36.555497 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.555459 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:36.555707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.555514 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:36.805072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.804374 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:36.805072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.804591 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:36.805072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:36.804646 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:37.120388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.120288 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:37.128484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.128448 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:37.129462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.129440 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:37.370852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.370755 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:37.469771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.469732 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerStarted","Data":"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d"} Oct 30 05:45:37.566441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.566391 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" start-of-body= Oct 30 05:45:37.566656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.566475 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": dial tcp 10.133.0.48:15021: connect: connection refused" Oct 30 05:45:37.812843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.812801 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:37.813179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:37.813143 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:38.386070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.384295 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:38.389212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.389173 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:45:38.389508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.389458 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 05:45:38.560186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.560156 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:38.561650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.561626 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:45:38.805073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.804447 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:38.805073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:38.804526 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:39.389070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:39.384482 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:45:39.389070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:39.384561 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 05:45:39.805718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:39.805671 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:39.806468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:39.806419 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:40.384937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:40.384477 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:45:40.384937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:40.384540 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 05:45:40.805526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:40.805484 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:40.805839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:40.805809 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:41.388070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:41.385759 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:45:41.388070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:41.385822 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 05:45:41.805211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:41.805169 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" start-of-body= Oct 30 05:45:41.805551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:41.805519 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": dial tcp 10.133.0.46:15021: connect: connection refused" Oct 30 05:45:42.384554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:42.384511 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" start-of-body= Oct 30 05:45:42.385223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:42.385190 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": dial tcp 10.133.0.47:15021: connect: connection refused" Oct 30 05:45:42.809017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:42.808986 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:42.810061 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:42.810022 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 05:45:43.388953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:43.388922 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:43.389461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:43.389445 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:43.390345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:43.390324 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 05:45:48.141700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.141668 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 05:45:48.142241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142226 6171 cpu_manager.go:414] "containerMap: removing container" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.142292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142243 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.142292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142255 6171 cpu_manager.go:414] "containerMap: removing container" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.142292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142260 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.142388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142358 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.142388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.142370 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:45:48.147146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.147124 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.165449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.165414 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"kube-root-ca.crt\"" Oct 30 05:45:48.177558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.177507 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"openshift-service-ca.crt\"" Oct 30 05:45:48.177971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.177931 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"simulator-w5-a35835-kserve-self-signed-certs\"" Oct 30 05:45:48.178144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.178010 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"default-dockercfg-644ms\"" Oct 30 05:45:48.182334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.182307 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 05:45:48.299489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299452 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjlqx\" (UniqueName: \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.299716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299499 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.299716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.299920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299747 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.299920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299790 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.299920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.299872 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.400962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.400851 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.401878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.401850 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.402457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.402434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.402924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.401705 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.403073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.402354 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.403183 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.402820 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.403309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.403293 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.403901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.403879 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjlqx\" (UniqueName: \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.404422 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.404389 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.415939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.415871 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.416541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.416490 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.436397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.436180 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjlqx\" (UniqueName: \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\") pod \"simulator-w5-a35835-kserve-5c45bf874c-6vqgl\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.459679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.459647 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:48.766396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:48.766373 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 05:45:48.767593 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:45:48.767568 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf38e784f_0f80_48ac_a403_33f3254b3b4a.slice/crio-ff8d9d013dbb7b793f056c012c8702cfa866bf4549675b4b3d04192d4e6abadb WatchSource:0}: Error finding container ff8d9d013dbb7b793f056c012c8702cfa866bf4549675b4b3d04192d4e6abadb: Status 404 returned error can't find the container with id ff8d9d013dbb7b793f056c012c8702cfa866bf4549675b4b3d04192d4e6abadb Oct 30 05:45:49.589784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:49.589741 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" event={"ID":"f38e784f-0f80-48ac-a403-33f3254b3b4a","Type":"ContainerStarted","Data":"9bf916eb5df7e7548fa92a20ccd90bbbcaa8a1f495d14fb509758a72f77fd277"} Oct 30 05:45:49.590228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:49.589798 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" event={"ID":"f38e784f-0f80-48ac-a403-33f3254b3b4a","Type":"ContainerStarted","Data":"ff8d9d013dbb7b793f056c012c8702cfa866bf4549675b4b3d04192d4e6abadb"} Oct 30 05:45:49.631572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:49.631535 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:45:49.642395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:49.642343 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podStartSLOduration=1.642328553 podStartE2EDuration="1.642328553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:49.631271893 +0000 UTC m=+808.071649068" watchObservedRunningTime="2026-10-30 05:45:49.642328553 +0000 UTC m=+808.082705747" Oct 30 05:45:50.460423 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:50.460363 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:56.381638 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:45:56.381599 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:45:58.460096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:58.460059 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:58.473085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:58.473028 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:45:58.679495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:45:58.679463 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:46:03.014723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:03.014637 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:46:04.374319 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.374283 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 05:46:04.375169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.375149 6171 cpu_manager.go:414] "containerMap: removing container" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:46:04.375231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.375173 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:46:04.375387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.375376 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="abbdec48-c0b0-4e11-b870-88f7ef54538b" containerName="cleanup" Oct 30 05:46:04.380826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.380806 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.398788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.398758 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 05:46:04.407951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.407920 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c78066\"/\"kube-root-ca.crt\"" Oct 30 05:46:04.407951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.407921 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c78066\"/\"openshift-service-ca.crt\"" Oct 30 05:46:04.408277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.408257 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c78066\"/\"premium-w2-c78066-kserve-self-signed-certs\"" Oct 30 05:46:04.409877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.409859 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c78066\"/\"default-dockercfg-bd499\"" Oct 30 05:46:04.482234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482197 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.482447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482243 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.482447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482356 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc66d\" (UniqueName: \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.482447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482407 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.482853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482466 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.482853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.482547 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.588737 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.588879 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.588924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.589001 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc66d\" (UniqueName: \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.589067 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.589487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.589095 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.594068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.589650 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.594068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.589937 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.594068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.590216 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.603233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.603201 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.604657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.604624 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.621159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.620008 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc66d\" (UniqueName: \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\") pod \"premium-w2-c78066-kserve-7c775786fd-zl65n\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:04.695664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:04.695637 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:05.004005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:05.003977 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 05:46:05.004948 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:46:05.004920 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf6056add_7ae7_4e66_98cb_105e8c75b4e7.slice/crio-ae41e410add30d99b009cafbfa05a975dada4da4cb743f6b13c931f37365d642 WatchSource:0}: Error finding container ae41e410add30d99b009cafbfa05a975dada4da4cb743f6b13c931f37365d642: Status 404 returned error can't find the container with id ae41e410add30d99b009cafbfa05a975dada4da4cb743f6b13c931f37365d642 Oct 30 05:46:05.708384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:05.708352 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" event={"ID":"f6056add-7ae7-4e66-98cb-105e8c75b4e7","Type":"ContainerStarted","Data":"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9"} Oct 30 05:46:05.708384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:05.708387 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" event={"ID":"f6056add-7ae7-4e66-98cb-105e8c75b4e7","Type":"ContainerStarted","Data":"ae41e410add30d99b009cafbfa05a975dada4da4cb743f6b13c931f37365d642"} Oct 30 05:46:05.751936 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:05.751887 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podStartSLOduration=1.751873932 podStartE2EDuration="1.751873932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:05.750847253 +0000 UTC m=+824.191224445" watchObservedRunningTime="2026-10-30 05:46:05.751873932 +0000 UTC m=+824.192251125" Oct 30 05:46:06.696445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:06.696400 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:14.696764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:14.696716 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:14.710109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:14.710080 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:14.789440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:14.789402 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:46:18.529076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:18.529021 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:46:21.320857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.320817 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 05:46:21.325957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.325937 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.356407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.356338 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"openshift-service-ca.crt\"" Oct 30 05:46:21.356407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.356381 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"kube-root-ca.crt\"" Oct 30 05:46:21.356721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.356706 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"premium-w4-e2e3d0-kserve-self-signed-certs\"" Oct 30 05:46:21.356721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.356706 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"default-dockercfg-l9xmp\"" Oct 30 05:46:21.359540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.359512 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 05:46:21.483348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.483546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483390 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv765\" (UniqueName: \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.483546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483451 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.483546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483505 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.483655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483544 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.483655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.483622 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585248 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585371 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585415 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv765\" (UniqueName: \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585446 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585473 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585509 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585789 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585852 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.585929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.585904 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.587766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.587743 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.588027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.588008 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.609276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.609244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv765\" (UniqueName: \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\") pod \"premium-w4-e2e3d0-kserve-768877854c-hs8k4\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:21.652909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.652879 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-e2e3d0\"/\"default-dockercfg-l9xmp\"" Oct 30 05:46:21.668086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:21.666660 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:22.010436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:22.010402 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 05:46:22.012195 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:46:22.012168 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod250b3345_4955_4a10_8afc_337ddd8b8b1e.slice/crio-573efdb68e51118c73513108214d24f115a2415783a254dc62c3e2148ff07e6b WatchSource:0}: Error finding container 573efdb68e51118c73513108214d24f115a2415783a254dc62c3e2148ff07e6b: Status 404 returned error can't find the container with id 573efdb68e51118c73513108214d24f115a2415783a254dc62c3e2148ff07e6b Oct 30 05:46:22.837460 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:22.837418 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" event={"ID":"250b3345-4955-4a10-8afc-337ddd8b8b1e","Type":"ContainerStarted","Data":"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d"} Oct 30 05:46:22.837460 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:22.837463 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" event={"ID":"250b3345-4955-4a10-8afc-337ddd8b8b1e","Type":"ContainerStarted","Data":"573efdb68e51118c73513108214d24f115a2415783a254dc62c3e2148ff07e6b"} Oct 30 05:46:22.879789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:22.879732 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podStartSLOduration=1.8797157389999999 podStartE2EDuration="1.879715739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:22.878206537 +0000 UTC m=+841.318583730" watchObservedRunningTime="2026-10-30 05:46:22.879715739 +0000 UTC m=+841.320092997" Oct 30 05:46:23.673402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:23.673370 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:26.380660 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:46:26.380623 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:46:30.349413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:30.349373 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:46:31.671631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:31.671600 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:31.681946 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:31.681914 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:31.906161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:31.906131 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:46:34.136205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.136170 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 05:46:34.140794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.140769 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.164716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.164685 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 05:46:34.172427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.172383 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"premium-w5-a35835-kserve-self-signed-certs\"" Oct 30 05:46:34.230150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230110 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.230332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230218 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.230332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230309 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.230409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230334 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.230479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230457 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.230514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.230488 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp99c\" (UniqueName: \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.332890 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.338954 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.339062 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.339089 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp99c\" (UniqueName: \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.339184 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.339382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.339270 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.340424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.340376 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.340657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.338146 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.341170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.341143 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.356086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.356020 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.367529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.367494 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.381434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.381399 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp99c\" (UniqueName: \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\") pod \"premium-w5-a35835-kserve-647d55d48b-s6fzx\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.453856 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.453765 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:34.784246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.784212 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 05:46:34.787686 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:46:34.787656 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd955cd_c784_49f4_be56_f4ac59582632.slice/crio-f2407a00a99cb926bbe194acc761bfd48cd0a2fc4b2a20946937fd32f4d3076a WatchSource:0}: Error finding container f2407a00a99cb926bbe194acc761bfd48cd0a2fc4b2a20946937fd32f4d3076a: Status 404 returned error can't find the container with id f2407a00a99cb926bbe194acc761bfd48cd0a2fc4b2a20946937fd32f4d3076a Oct 30 05:46:34.919619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.919580 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" event={"ID":"3dd955cd-c784-49f4-be56-f4ac59582632","Type":"ContainerStarted","Data":"6561fb6a0bf2e14303dd7191d446e5e5417a8b4e33fae51cb599e6b8fc6280dc"} Oct 30 05:46:34.919793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.919624 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" event={"ID":"3dd955cd-c784-49f4-be56-f4ac59582632","Type":"ContainerStarted","Data":"f2407a00a99cb926bbe194acc761bfd48cd0a2fc4b2a20946937fd32f4d3076a"} Oct 30 05:46:34.942455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.942416 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 05:46:34.946829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.946803 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:34.961853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.961830 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-197e9b-openshift-default-dockercfg-w5q2q\"" Oct 30 05:46:34.973527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.973475 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podStartSLOduration=0.973462366 podStartE2EDuration="973.462366ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:34.970025898 +0000 UTC m=+853.410403094" watchObservedRunningTime="2026-10-30 05:46:34.973462366 +0000 UTC m=+853.413839558" Oct 30 05:46:34.988253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:34.988221 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 05:46:35.045627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045531 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045584 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbhfv\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045672 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045734 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045892 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.045990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.045958 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.046039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.046027 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.046092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.046078 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147024 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147103 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147181 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147208 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147248 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147273 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbhfv\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147304 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147346 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147373 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.147583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147473 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.148265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147658 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.148265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147724 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.148265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.147790 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.148265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.148022 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.149693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.149673 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.149946 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.149929 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.171903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.171872 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.172071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.172034 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbhfv\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.260692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.260651 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:35.454222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.454185 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:35.589375 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.589348 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 05:46:35.590615 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:46:35.590589 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f5320e_e28b_46eb_b2d8_6f63e83f2274.slice/crio-c2d5f790aa35bb961562042cc5b3577db48f8845450e3437f2e922ce56248c29 WatchSource:0}: Error finding container c2d5f790aa35bb961562042cc5b3577db48f8845450e3437f2e922ce56248c29: Status 404 returned error can't find the container with id c2d5f790aa35bb961562042cc5b3577db48f8845450e3437f2e922ce56248c29 Oct 30 05:46:35.593230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.593196 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:35.593312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.593264 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:35.593312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.593295 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:35.927664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.927557 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" event={"ID":"e2f5320e-e28b-46eb-b2d8-6f63e83f2274","Type":"ContainerStarted","Data":"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89"} Oct 30 05:46:35.927664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.927626 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" event={"ID":"e2f5320e-e28b-46eb-b2d8-6f63e83f2274","Type":"ContainerStarted","Data":"c2d5f790aa35bb961562042cc5b3577db48f8845450e3437f2e922ce56248c29"} Oct 30 05:46:35.973949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:35.973889 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podStartSLOduration=1.973872075 podStartE2EDuration="1.973872075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:35.97239376 +0000 UTC m=+854.412770946" watchObservedRunningTime="2026-10-30 05:46:35.973872075 +0000 UTC m=+854.414249268" Oct 30 05:46:36.261365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:36.261326 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:36.262672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:36.262646 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:36.262812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:36.262703 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:37.261164 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:37.261125 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:37.261361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:37.261188 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:38.261462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:38.261425 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:38.261881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:38.261503 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:39.261117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:39.260958 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:39.261117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:39.261020 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:40.262170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:40.262123 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:40.262713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:40.262218 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:41.261488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:41.261453 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:41.261684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:41.261511 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:42.261631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:42.261590 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" start-of-body= Oct 30 05:46:42.262139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:42.261689 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": dial tcp 10.133.0.65:15021: connect: connection refused" Oct 30 05:46:43.266807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:43.266776 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:43.267344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:43.267230 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:43.267979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:43.267949 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 05:46:44.454415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:44.454377 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:44.466782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:44.466752 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:45.006251 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:45.006223 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:46:50.274877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.274841 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 05:46:50.279342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.279319 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.309543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.309442 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c78066-openshift-default-dockercfg-g958x\"" Oct 30 05:46:50.313669 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.313635 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 05:46:50.369118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369084 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369188 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369213 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h26kw\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369283 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369347 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369369 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369477 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369533 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.369670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.369646 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:46:50.470514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.470478 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.470810 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.470779 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.470942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.470824 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471099 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471149 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471202 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471277 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471331 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.471498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.471357 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h26kw\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.472034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.472014 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.473248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.473222 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.473401 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.473380 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.473494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.473435 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.474270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.474244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.475124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.475101 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.475440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.475417 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.500249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.500221 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.501351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.501319 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h26kw\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.593441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.593360 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:50.930643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.930612 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 05:46:50.930770 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:46:50.930670 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb85c8d_fe38_46bd_9eb2_8f3e37d84c73.slice/crio-af24ec0095207ad42bb40b4b6fc9afd9c57c6d81b8d15fddcf39cb7166ad3f56 WatchSource:0}: Error finding container af24ec0095207ad42bb40b4b6fc9afd9c57c6d81b8d15fddcf39cb7166ad3f56: Status 404 returned error can't find the container with id af24ec0095207ad42bb40b4b6fc9afd9c57c6d81b8d15fddcf39cb7166ad3f56 Oct 30 05:46:50.933355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.933322 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:50.933431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.933386 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:50.933431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:50.933421 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:46:51.032062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.032018 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" event={"ID":"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73","Type":"ContainerStarted","Data":"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1"} Oct 30 05:46:51.032248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.032104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" event={"ID":"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73","Type":"ContainerStarted","Data":"af24ec0095207ad42bb40b4b6fc9afd9c57c6d81b8d15fddcf39cb7166ad3f56"} Oct 30 05:46:51.087559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.087492 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podStartSLOduration=1.087471383 podStartE2EDuration="1.087471383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:51.085454883 +0000 UTC m=+869.525832080" watchObservedRunningTime="2026-10-30 05:46:51.087471383 +0000 UTC m=+869.527848577" Oct 30 05:46:51.596070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.594160 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:51.734526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.734490 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2"] Oct 30 05:46:51.738594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.738569 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.759232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.759194 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:46:51.759232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.759333 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:51.759232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.760712 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:51.759232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.760760 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:51.762939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.762913 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2"] Oct 30 05:46:51.803281 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.803237 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.822269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.822215 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.822469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.822357 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4mz6\" (UniqueName: \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.924465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.924348 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.924465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.924449 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.924729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.924546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4mz6\" (UniqueName: \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.925517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.925488 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.930569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.928608 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:51.961415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:51.959396 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4mz6\" (UniqueName: \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\") pod \"maas-api-revoke-sub-c8360874-ph9p2\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:52.055314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:52.055280 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:52.421073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:52.414828 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2"] Oct 30 05:46:52.600370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:52.600333 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:52.602821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:52.602781 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:53.059998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:53.059960 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" event={"ID":"5e8e42ef-89a8-47d7-a6ae-350e4f044b30","Type":"ContainerStarted","Data":"65a92e7972ef8e832edef10e1a28b7a5876414c60f83db080cd99a21ebba040a"} Oct 30 05:46:53.060217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:53.060016 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" event={"ID":"5e8e42ef-89a8-47d7-a6ae-350e4f044b30","Type":"ContainerStarted","Data":"4681c3a3925add74ddecb3fe7bda03303ce5177d8f8a2c87db0c0bc04102895c"} Oct 30 05:46:53.106469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:53.106352 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" podStartSLOduration=2.106335301 podStartE2EDuration="2.106335301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:53.1062605 +0000 UTC m=+871.546637693" watchObservedRunningTime="2026-10-30 05:46:53.106335301 +0000 UTC m=+871.546712494" Oct 30 05:46:53.605085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:53.601521 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:53.605085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:53.601589 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:54.068694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:54.068660 6171 generic.go:356] "Generic (PLEG): container finished" podID="5e8e42ef-89a8-47d7-a6ae-350e4f044b30" containerID="65a92e7972ef8e832edef10e1a28b7a5876414c60f83db080cd99a21ebba040a" exitCode=0 Oct 30 05:46:54.068897 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:54.068710 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" event={"ID":"5e8e42ef-89a8-47d7-a6ae-350e4f044b30","Type":"ContainerDied","Data":"65a92e7972ef8e832edef10e1a28b7a5876414c60f83db080cd99a21ebba040a"} Oct 30 05:46:54.605072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:54.604897 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:54.605072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:54.604958 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:55.274133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.274106 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:55.378106 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.377974 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\") pod \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " Oct 30 05:46:55.378106 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.378106 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\" (UniqueName: \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\") pod \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " Oct 30 05:46:55.378309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.378135 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\") pod \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\" (UID: \"5e8e42ef-89a8-47d7-a6ae-350e4f044b30\") " Oct 30 05:46:55.378557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.378524 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca" pod "5e8e42ef-89a8-47d7-a6ae-350e4f044b30" (UID: "5e8e42ef-89a8-47d7-a6ae-350e4f044b30"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:55.397251 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.397205 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca" pod "5e8e42ef-89a8-47d7-a6ae-350e4f044b30" (UID: "5e8e42ef-89a8-47d7-a6ae-350e4f044b30"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:55.405119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.402183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6" pod "5e8e42ef-89a8-47d7-a6ae-350e4f044b30" (UID: "5e8e42ef-89a8-47d7-a6ae-350e4f044b30"). InnerVolumeSpecName "kube-api-access-j4mz6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:55.479740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.479702 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4mz6\" (UniqueName: \"kubernetes.io/projected/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-kube-api-access-j4mz6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:46:55.479740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.479732 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:46:55.479740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.479744 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e8e42ef-89a8-47d7-a6ae-350e4f044b30-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:46:55.604082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.603487 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:55.604082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:55.603551 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:56.107207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:56.107116 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" Oct 30 05:46:56.107207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:56.107158 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2" event={"ID":"5e8e42ef-89a8-47d7-a6ae-350e4f044b30","Type":"ContainerDied","Data":"4681c3a3925add74ddecb3fe7bda03303ce5177d8f8a2c87db0c0bc04102895c"} Oct 30 05:46:56.107628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:56.107220 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4681c3a3925add74ddecb3fe7bda03303ce5177d8f8a2c87db0c0bc04102895c" Oct 30 05:46:56.388707 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:46:56.388558 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:46:56.605295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:56.605254 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:56.605521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:56.605328 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:57.594298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:57.594258 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 05:46:57.594721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:57.594321 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 05:46:58.598620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:58.598587 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:58.599032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:58.598866 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:46:58.599580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:46:58.599558 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 05:47:05.933722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.933686 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:47:05.934453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.934267 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5e8e42ef-89a8-47d7-a6ae-350e4f044b30" containerName="revoke-keys" Oct 30 05:47:05.934453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.934280 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e8e42ef-89a8-47d7-a6ae-350e4f044b30" containerName="revoke-keys" Oct 30 05:47:05.934453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.934398 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e8e42ef-89a8-47d7-a6ae-350e4f044b30" containerName="revoke-keys" Oct 30 05:47:05.937627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.937610 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:05.957746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.957721 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-128436-openshift-default-dockercfg-dwwj9\"" Oct 30 05:47:05.982435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:05.982402 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:47:06.068667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.068626 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.068863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.068679 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.068863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.068800 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.068863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.068856 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.068987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.068953 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.069025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.069006 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.069095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.069035 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.069156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.069144 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.069196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.069177 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb44h\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170084 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170145 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170180 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170200 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170220 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170291 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170325 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb44h\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170373 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170423 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170737 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170633 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170803 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170782 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.170902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.170878 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.171090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.171066 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.171182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.171103 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.172993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.172968 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.173283 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.173264 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.192300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.192227 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8"] Oct 30 05:47:06.196451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.196434 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.199889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.199859 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb44h\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.200019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.199940 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.217141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.217114 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:06.217141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.217137 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:47:06.220862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.220837 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:06.221017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.220998 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:06.224702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.224679 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8"] Oct 30 05:47:06.251926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.251889 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:06.271770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.271737 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.271900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.271801 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.271900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.271892 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4cd5\" (UniqueName: \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.373653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.373611 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.373855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.373680 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.373855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.373721 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4cd5\" (UniqueName: \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.374588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.374543 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.376413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.376390 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.397925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.397888 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4cd5\" (UniqueName: \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\") pod \"maas-api-revoke-sub-968d7f55-8pws8\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.523297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.523261 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:06.590628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.590549 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:47:06.595066 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:06.595012 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f605e3c_ee03_49ba_b55d_f999c2c6684b.slice/crio-2bd140deb3f01700bbed4fe31ccfdc66079c332ac4033e6096f15738e456d09f WatchSource:0}: Error finding container 2bd140deb3f01700bbed4fe31ccfdc66079c332ac4033e6096f15738e456d09f: Status 404 returned error can't find the container with id 2bd140deb3f01700bbed4fe31ccfdc66079c332ac4033e6096f15738e456d09f Oct 30 05:47:06.599085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.598994 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:06.599212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.599161 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:06.599278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.599210 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:06.839772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:06.839743 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8"] Oct 30 05:47:06.840729 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:06.840703 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d3e4d9_7217_448d_9542_c41032805a57.slice/crio-67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590 WatchSource:0}: Error finding container 67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590: Status 404 returned error can't find the container with id 67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590 Oct 30 05:47:07.213117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.213076 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" event={"ID":"78d3e4d9-7217-448d-9542-c41032805a57","Type":"ContainerStarted","Data":"fc568c0fe26e496543e6e2523a170c18e5f34ff35a00bb8ad835c3f5b479035a"} Oct 30 05:47:07.213633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.213146 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" event={"ID":"78d3e4d9-7217-448d-9542-c41032805a57","Type":"ContainerStarted","Data":"67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590"} Oct 30 05:47:07.215377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.215354 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" event={"ID":"1f605e3c-ee03-49ba-b55d-f999c2c6684b","Type":"ContainerStarted","Data":"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808"} Oct 30 05:47:07.215504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.215397 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" event={"ID":"1f605e3c-ee03-49ba-b55d-f999c2c6684b","Type":"ContainerStarted","Data":"2bd140deb3f01700bbed4fe31ccfdc66079c332ac4033e6096f15738e456d09f"} Oct 30 05:47:07.252349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.252313 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:07.254262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.254233 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.254412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.254281 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:07.256284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.256242 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" podStartSLOduration=1.256228295 podStartE2EDuration="1.256228295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:07.254148949 +0000 UTC m=+885.694526143" watchObservedRunningTime="2026-10-30 05:47:07.256228295 +0000 UTC m=+885.696605503" Oct 30 05:47:07.355528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.355422 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podStartSLOduration=2.355405886 podStartE2EDuration="2.355405886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:07.354925212 +0000 UTC m=+885.795302405" watchObservedRunningTime="2026-10-30 05:47:07.355405886 +0000 UTC m=+885.795783148" Oct 30 05:47:07.546294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.546260 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk"] Oct 30 05:47:07.550318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.550300 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.612381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.612276 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk"] Oct 30 05:47:07.620608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.613313 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.620608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.613428 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.620608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.613480 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-np5tm\" (UniqueName: \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.714464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.714423 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.714643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.714552 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.714643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.714634 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-np5tm\" (UniqueName: \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.715250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.715224 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.717093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.717072 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.738869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.738838 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-np5tm\" (UniqueName: \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\") pod \"maas-api-revoke-sub-1c771a3e-qx6bk\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:07.864448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:07.864320 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:08.213310 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.213278 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk"] Oct 30 05:47:08.213765 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:08.213743 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7941aa2f_2fcc_49ac_95b2_3163c6c0f0f0.slice/crio-7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a WatchSource:0}: Error finding container 7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a: Status 404 returned error can't find the container with id 7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a Oct 30 05:47:08.222806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.222773 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" event={"ID":"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0","Type":"ContainerStarted","Data":"7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a"} Oct 30 05:47:08.224857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.224827 6171 generic.go:356] "Generic (PLEG): container finished" podID="78d3e4d9-7217-448d-9542-c41032805a57" containerID="fc568c0fe26e496543e6e2523a170c18e5f34ff35a00bb8ad835c3f5b479035a" exitCode=0 Oct 30 05:47:08.225380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.225347 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" event={"ID":"78d3e4d9-7217-448d-9542-c41032805a57","Type":"ContainerDied","Data":"fc568c0fe26e496543e6e2523a170c18e5f34ff35a00bb8ad835c3f5b479035a"} Oct 30 05:47:08.252918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.252881 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.253133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.252937 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:08.354388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.354355 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw"] Oct 30 05:47:08.359847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.359825 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.379251 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.379223 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw"] Oct 30 05:47:08.425911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.425875 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.425911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.425915 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.426177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.426081 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngzjf\" (UniqueName: \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.527429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.527397 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngzjf\" (UniqueName: \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.527572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.527504 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.527572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.527527 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.528197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.528172 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.530126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.530092 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.561390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.561363 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngzjf\" (UniqueName: \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\") pod \"maas-api-revoke-sub-fa02d109-gjphw\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:08.673890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:08.673852 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:09.009121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.009098 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw"] Oct 30 05:47:09.011206 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:09.011173 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3785394b_1444_45c6_a068_04ceebee95c9.slice/crio-d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad WatchSource:0}: Error finding container d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad: Status 404 returned error can't find the container with id d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad Oct 30 05:47:09.237002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.236800 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" event={"ID":"3785394b-1444-45c6-a068-04ceebee95c9","Type":"ContainerStarted","Data":"d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad"} Oct 30 05:47:09.242782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.242751 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" event={"ID":"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0","Type":"ContainerStarted","Data":"c1f1d5353ce692fef76aa24bdf0836a38ece49aa43cec8493aff425662803ef7"} Oct 30 05:47:09.252469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.252397 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:09.252469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.252461 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:09.308201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.305174 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" podStartSLOduration=2.305149067 podStartE2EDuration="2.305149067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:09.292343213 +0000 UTC m=+887.732720407" watchObservedRunningTime="2026-10-30 05:47:09.305149067 +0000 UTC m=+887.745526261" Oct 30 05:47:09.522567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.522536 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:09.549486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.549454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\") pod \"78d3e4d9-7217-448d-9542-c41032805a57\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " Oct 30 05:47:09.549631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.549616 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\" (UniqueName: \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\") pod \"78d3e4d9-7217-448d-9542-c41032805a57\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " Oct 30 05:47:09.549693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.549640 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\") pod \"78d3e4d9-7217-448d-9542-c41032805a57\" (UID: \"78d3e4d9-7217-448d-9542-c41032805a57\") " Oct 30 05:47:09.550141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.550114 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca" pod "78d3e4d9-7217-448d-9542-c41032805a57" (UID: "78d3e4d9-7217-448d-9542-c41032805a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:09.551832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.551803 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca" pod "78d3e4d9-7217-448d-9542-c41032805a57" (UID: "78d3e4d9-7217-448d-9542-c41032805a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:09.551832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.551815 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5" pod "78d3e4d9-7217-448d-9542-c41032805a57" (UID: "78d3e4d9-7217-448d-9542-c41032805a57"). InnerVolumeSpecName "kube-api-access-g4cd5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:09.651267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.651235 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4cd5\" (UniqueName: \"kubernetes.io/projected/78d3e4d9-7217-448d-9542-c41032805a57-kube-api-access-g4cd5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:09.651267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.651267 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:09.651490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:09.651277 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d3e4d9-7217-448d-9542-c41032805a57-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:10.250948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.250909 6171 generic.go:356] "Generic (PLEG): container finished" podID="3785394b-1444-45c6-a068-04ceebee95c9" containerID="000fba00a7019259d25c1afc1fda15881dfcf34fab8286a5544a7ba548175eb9" exitCode=0 Oct 30 05:47:10.251427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.250945 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" event={"ID":"3785394b-1444-45c6-a068-04ceebee95c9","Type":"ContainerDied","Data":"000fba00a7019259d25c1afc1fda15881dfcf34fab8286a5544a7ba548175eb9"} Oct 30 05:47:10.252364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.252340 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:10.252508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.252392 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:10.252564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.252496 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" event={"ID":"78d3e4d9-7217-448d-9542-c41032805a57","Type":"ContainerDied","Data":"67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590"} Oct 30 05:47:10.252564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.252535 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8" Oct 30 05:47:10.252709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.252539 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67eb6929b969da637c0b805aeae5cef03dcb2af98b190aca5353280ccc726590" Oct 30 05:47:10.254155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.254132 6171 generic.go:356] "Generic (PLEG): container finished" podID="7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" containerID="c1f1d5353ce692fef76aa24bdf0836a38ece49aa43cec8493aff425662803ef7" exitCode=0 Oct 30 05:47:10.254271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:10.254202 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" event={"ID":"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0","Type":"ContainerDied","Data":"c1f1d5353ce692fef76aa24bdf0836a38ece49aa43cec8493aff425662803ef7"} Oct 30 05:47:11.252619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.252585 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:11.253027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.252642 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:11.467840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.467801 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:11.519717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.519656 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:11.575809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575774 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\") pod \"3785394b-1444-45c6-a068-04ceebee95c9\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " Oct 30 05:47:11.575809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575815 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\") pod \"3785394b-1444-45c6-a068-04ceebee95c9\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " Oct 30 05:47:11.576021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575910 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\" (UniqueName: \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\") pod \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " Oct 30 05:47:11.576021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575930 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\") pod \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " Oct 30 05:47:11.576021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575975 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\") pod \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\" (UID: \"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0\") " Oct 30 05:47:11.576021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.575992 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\" (UniqueName: \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\") pod \"3785394b-1444-45c6-a068-04ceebee95c9\" (UID: \"3785394b-1444-45c6-a068-04ceebee95c9\") " Oct 30 05:47:11.576540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.576336 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca" pod "3785394b-1444-45c6-a068-04ceebee95c9" (UID: "3785394b-1444-45c6-a068-04ceebee95c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:11.576540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.576504 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca" pod "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" (UID: "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:11.576625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.576608 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:11.576666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.576623 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:11.578686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.578660 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm" pod "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" (UID: "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0"). InnerVolumeSpecName "kube-api-access-np5tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:11.578881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.578860 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca" pod "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" (UID: "7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:11.578947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.578933 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf" pod "3785394b-1444-45c6-a068-04ceebee95c9" (UID: "3785394b-1444-45c6-a068-04ceebee95c9"). InnerVolumeSpecName "kube-api-access-ngzjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:11.579031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.579012 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca" pod "3785394b-1444-45c6-a068-04ceebee95c9" (UID: "3785394b-1444-45c6-a068-04ceebee95c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:11.677522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.677433 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3785394b-1444-45c6-a068-04ceebee95c9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:11.677522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.677468 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-np5tm\" (UniqueName: \"kubernetes.io/projected/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-kube-api-access-np5tm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:11.677522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.677483 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:11.677522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:11.677497 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngzjf\" (UniqueName: \"kubernetes.io/projected/3785394b-1444-45c6-a068-04ceebee95c9-kube-api-access-ngzjf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:12.252762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.252722 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:12.253190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.252787 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:12.268710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.268679 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" Oct 30 05:47:12.268883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.268707 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk" event={"ID":"7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0","Type":"ContainerDied","Data":"7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a"} Oct 30 05:47:12.268883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.268762 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7541edba378ae3c5c57b8bf565f7b81897e50b490e3cb289ac0ed55a72b5618a" Oct 30 05:47:12.270285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.270258 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" Oct 30 05:47:12.270411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.270312 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw" event={"ID":"3785394b-1444-45c6-a068-04ceebee95c9","Type":"ContainerDied","Data":"d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad"} Oct 30 05:47:12.270411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.270379 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d814dcc760ab6503c2d98e8f482660566bdce006467774e43ae6b4ea9b1219ad" Oct 30 05:47:12.627178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:12.627089 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:47:13.252481 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:13.252443 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body= Oct 30 05:47:13.252668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:13.252505 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" Oct 30 05:47:15.237265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:15.237227 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:15.238000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:15.237976 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:15.242705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:15.242682 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:47:18.748790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.748755 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2"] Oct 30 05:47:18.749355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749339 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" containerName="revoke-keys" Oct 30 05:47:18.749355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749355 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" containerName="revoke-keys" Oct 30 05:47:18.749476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749382 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78d3e4d9-7217-448d-9542-c41032805a57" containerName="revoke-keys" Oct 30 05:47:18.749476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749388 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d3e4d9-7217-448d-9542-c41032805a57" containerName="revoke-keys" Oct 30 05:47:18.749476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749402 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3785394b-1444-45c6-a068-04ceebee95c9" containerName="revoke-keys" Oct 30 05:47:18.749476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749408 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3785394b-1444-45c6-a068-04ceebee95c9" containerName="revoke-keys" Oct 30 05:47:18.749683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749510 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" containerName="revoke-keys" Oct 30 05:47:18.749683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749521 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d3e4d9-7217-448d-9542-c41032805a57" containerName="revoke-keys" Oct 30 05:47:18.749683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.749527 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3785394b-1444-45c6-a068-04ceebee95c9" containerName="revoke-keys" Oct 30 05:47:18.754067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.754019 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.774225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.774198 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2"] Oct 30 05:47:18.782554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.782515 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:18.782732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.782543 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:18.782824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.782582 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:18.784964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.784946 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:47:18.845511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.845470 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.845658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.845542 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.845658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.845625 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pz97m\" (UniqueName: \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.946885 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.946849 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.947113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.946912 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.947113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.946940 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pz97m\" (UniqueName: \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.947713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.947694 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.949395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.949379 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:18.970654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:18.970623 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pz97m\" (UniqueName: \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\") pod \"maas-api-revoke-sub-449bb3e2-vffs2\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:19.067485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:19.067403 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:19.372358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:19.372316 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2"] Oct 30 05:47:19.373620 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:19.373595 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6e29ce9_d30b_4e39_b0d6_a62293f929c8.slice/crio-2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94 WatchSource:0}: Error finding container 2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94: Status 404 returned error can't find the container with id 2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94 Oct 30 05:47:19.996382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:19.996354 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:47:20.000962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.000940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.035641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.035611 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:47:20.160077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.159958 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160009 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160162 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160216 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160307 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78b8f\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160348 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160399 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160511 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.160584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.160548 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261571 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261547 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261576 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261595 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261782 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261834 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261891 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78b8f\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.261994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.261940 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262292 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262322 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262528 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262587 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.262651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.262608 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.264038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.264017 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.264829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.264806 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.290022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.289993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.290187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.289993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78b8f\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.320226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.320198 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:20.342905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.342875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" event={"ID":"d6e29ce9-d30b-4e39-b0d6-a62293f929c8","Type":"ContainerStarted","Data":"069837a22f0b80fb6f44be855a0cb97197047a26abefc9a52cde5dee59207369"} Oct 30 05:47:20.343010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.342939 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" event={"ID":"d6e29ce9-d30b-4e39-b0d6-a62293f929c8","Type":"ContainerStarted","Data":"2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94"} Oct 30 05:47:20.387380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.387322 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" podStartSLOduration=2.387305528 podStartE2EDuration="2.387305528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:20.384945895 +0000 UTC m=+898.825323089" watchObservedRunningTime="2026-10-30 05:47:20.387305528 +0000 UTC m=+898.827682722" Oct 30 05:47:20.616154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.613294 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9"] Oct 30 05:47:20.622552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.622526 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.636742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.636721 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9"] Oct 30 05:47:20.654794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.654760 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:47:20.668078 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:20.665798 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63075e82_f898_431d_aca8_a6cea2a94cb7.slice/crio-b870b3da10993e51449a59af9307570d89878acdee9a367aa9dfa9cb1e8657a4 WatchSource:0}: Error finding container b870b3da10993e51449a59af9307570d89878acdee9a367aa9dfa9cb1e8657a4: Status 404 returned error can't find the container with id b870b3da10993e51449a59af9307570d89878acdee9a367aa9dfa9cb1e8657a4 Oct 30 05:47:20.668078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.667965 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:20.668078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.668071 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:20.668364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.668125 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:20.767965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.767937 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.768198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.768173 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkpkk\" (UniqueName: \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.768486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.768462 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.869449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.869359 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.869624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.869470 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.869624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.869520 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkpkk\" (UniqueName: \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.870807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.870782 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.872949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.872911 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.893173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.893153 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkpkk\" (UniqueName: \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\") pod \"maas-api-revoke-sub-e055c622-82sp9\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:20.938995 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:20.938967 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:21.247493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.247462 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9"] Oct 30 05:47:21.247995 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:21.247907 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8eb26885_8ce8_4af2_8736_606701a15d7e.slice/crio-f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794 WatchSource:0}: Error finding container f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794: Status 404 returned error can't find the container with id f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794 Oct 30 05:47:21.350507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.350467 6171 generic.go:356] "Generic (PLEG): container finished" podID="d6e29ce9-d30b-4e39-b0d6-a62293f929c8" containerID="069837a22f0b80fb6f44be855a0cb97197047a26abefc9a52cde5dee59207369" exitCode=0 Oct 30 05:47:21.350613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.350532 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" event={"ID":"d6e29ce9-d30b-4e39-b0d6-a62293f929c8","Type":"ContainerDied","Data":"069837a22f0b80fb6f44be855a0cb97197047a26abefc9a52cde5dee59207369"} Oct 30 05:47:21.351827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.351799 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" event={"ID":"8eb26885-8ce8-4af2-8736-606701a15d7e","Type":"ContainerStarted","Data":"f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794"} Oct 30 05:47:21.353470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.353447 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" event={"ID":"63075e82-f898-431d-aca8-a6cea2a94cb7","Type":"ContainerStarted","Data":"817fbf830bfd087ff4a222519ad23e313c402fc6f76be4627434d8accb61d571"} Oct 30 05:47:21.353570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.353491 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" event={"ID":"63075e82-f898-431d-aca8-a6cea2a94cb7","Type":"ContainerStarted","Data":"b870b3da10993e51449a59af9307570d89878acdee9a367aa9dfa9cb1e8657a4"} Oct 30 05:47:21.437312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:21.437258 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podStartSLOduration=2.437242144 podStartE2EDuration="2.437242144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:21.434881361 +0000 UTC m=+899.875258554" watchObservedRunningTime="2026-10-30 05:47:21.437242144 +0000 UTC m=+899.877619336" Oct 30 05:47:22.320656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.320627 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:22.322490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.322461 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:22.322621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.322510 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:22.370513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.370474 6171 generic.go:356] "Generic (PLEG): container finished" podID="8eb26885-8ce8-4af2-8736-606701a15d7e" containerID="512c9b22f77600f55aaa4bf2c0a36f4f7b2308a4bfaddc66ff38a320d4242ced" exitCode=0 Oct 30 05:47:22.370679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.370568 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" event={"ID":"8eb26885-8ce8-4af2-8736-606701a15d7e","Type":"ContainerDied","Data":"512c9b22f77600f55aaa4bf2c0a36f4f7b2308a4bfaddc66ff38a320d4242ced"} Oct 30 05:47:22.627384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.627354 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:22.712110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.710136 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5"] Oct 30 05:47:22.712110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.711102 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d6e29ce9-d30b-4e39-b0d6-a62293f929c8" containerName="revoke-keys" Oct 30 05:47:22.712110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.711123 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e29ce9-d30b-4e39-b0d6-a62293f929c8" containerName="revoke-keys" Oct 30 05:47:22.712110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.711270 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e29ce9-d30b-4e39-b0d6-a62293f929c8" containerName="revoke-keys" Oct 30 05:47:22.715347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.715322 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7"] Oct 30 05:47:22.715620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.715596 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.719723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.719704 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.739388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.739345 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7"] Oct 30 05:47:22.739767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.739707 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\" (UniqueName: \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\") pod \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " Oct 30 05:47:22.739900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.739864 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\") pod \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " Oct 30 05:47:22.740471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.740001 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\") pod \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\" (UID: \"d6e29ce9-d30b-4e39-b0d6-a62293f929c8\") " Oct 30 05:47:22.740471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.740348 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sctmn\" (UniqueName: \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.740471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.740401 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca" pod "d6e29ce9-d30b-4e39-b0d6-a62293f929c8" (UID: "d6e29ce9-d30b-4e39-b0d6-a62293f929c8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:22.740471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.740444 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.740752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.740523 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.741970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.741922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.742187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.742163 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7brh\" (UniqueName: \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.742300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.742244 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.742369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.742348 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:22.744407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.744375 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca" pod "d6e29ce9-d30b-4e39-b0d6-a62293f929c8" (UID: "d6e29ce9-d30b-4e39-b0d6-a62293f929c8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:22.744691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.744666 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m" pod "d6e29ce9-d30b-4e39-b0d6-a62293f929c8" (UID: "d6e29ce9-d30b-4e39-b0d6-a62293f929c8"). InnerVolumeSpecName "kube-api-access-pz97m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:22.744941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.744915 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5"] Oct 30 05:47:22.843444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843348 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sctmn\" (UniqueName: \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.843444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843417 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.843674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843460 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.843674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843538 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.843674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7brh\" (UniqueName: \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.843674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843627 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.843881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843710 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:22.843881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.843732 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pz97m\" (UniqueName: \"kubernetes.io/projected/d6e29ce9-d30b-4e39-b0d6-a62293f929c8-kube-api-access-pz97m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:22.844170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.844146 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.844170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.844157 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.846282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.846257 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.846353 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.846281 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:22.867130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.867097 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sctmn\" (UniqueName: \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\") pod \"maas-api-revoke-sub-e81cc108-7wkd7\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:22.867274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:22.867100 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7brh\" (UniqueName: \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\") pod \"maas-api-revoke-sub-c28447a3-8skr5\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:23.039105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.039072 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:23.074384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.074349 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:23.321653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.321609 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:23.322020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.321669 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:23.369876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.369841 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5"] Oct 30 05:47:23.370156 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:23.370036 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5751de18_3728_4198_a097_d149fbddbc61.slice/crio-0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3 WatchSource:0}: Error finding container 0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3: Status 404 returned error can't find the container with id 0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3 Oct 30 05:47:23.411281 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.411246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" event={"ID":"d6e29ce9-d30b-4e39-b0d6-a62293f929c8","Type":"ContainerDied","Data":"2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94"} Oct 30 05:47:23.411453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.411305 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b0d11c0632a0df855466f2a49bcb577a6c61be15914986dbeeffee44e4b9a94" Oct 30 05:47:23.411453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.411260 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2" Oct 30 05:47:23.412869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.412841 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" event={"ID":"5751de18-3728-4198-a097-d149fbddbc61","Type":"ContainerStarted","Data":"0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3"} Oct 30 05:47:23.424942 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:23.424913 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf3fb25_226b_41ab_9b75_b0bf81a864f1.slice/crio-eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c WatchSource:0}: Error finding container eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c: Status 404 returned error can't find the container with id eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c Oct 30 05:47:23.426462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.426441 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7"] Oct 30 05:47:23.719891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.719860 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:23.754851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.754818 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\") pod \"8eb26885-8ce8-4af2-8736-606701a15d7e\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " Oct 30 05:47:23.755038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.754921 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\" (UniqueName: \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\") pod \"8eb26885-8ce8-4af2-8736-606701a15d7e\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " Oct 30 05:47:23.755038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.754989 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\") pod \"8eb26885-8ce8-4af2-8736-606701a15d7e\" (UID: \"8eb26885-8ce8-4af2-8736-606701a15d7e\") " Oct 30 05:47:23.755397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.755362 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca" pod "8eb26885-8ce8-4af2-8736-606701a15d7e" (UID: "8eb26885-8ce8-4af2-8736-606701a15d7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:23.755699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.755648 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:23.757139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.757109 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca" pod "8eb26885-8ce8-4af2-8736-606701a15d7e" (UID: "8eb26885-8ce8-4af2-8736-606701a15d7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:23.757296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.757278 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk" pod "8eb26885-8ce8-4af2-8736-606701a15d7e" (UID: "8eb26885-8ce8-4af2-8736-606701a15d7e"). InnerVolumeSpecName "kube-api-access-vkpkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:23.856728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.856677 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8eb26885-8ce8-4af2-8736-606701a15d7e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:23.856728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:23.856709 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vkpkk\" (UniqueName: \"kubernetes.io/projected/8eb26885-8ce8-4af2-8736-606701a15d7e-kube-api-access-vkpkk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:24.323702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.323654 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:24.324268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.324239 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:24.424273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.424231 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" event={"ID":"8eb26885-8ce8-4af2-8736-606701a15d7e","Type":"ContainerDied","Data":"f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794"} Oct 30 05:47:24.424439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.424300 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f060f8f0226d7b202a687ac52008da04a4e87780aaf53a3db3d61a018d0e3794" Oct 30 05:47:24.424646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.424628 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9" Oct 30 05:47:24.426459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.426426 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" event={"ID":"5751de18-3728-4198-a097-d149fbddbc61","Type":"ContainerStarted","Data":"b174338d459b16d246019f87f7c18d8c87e2700bb0fe0e1fd43bc01cfcc5d392"} Oct 30 05:47:24.429373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.429341 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" event={"ID":"dcf3fb25-226b-41ab-9b75-b0bf81a864f1","Type":"ContainerStarted","Data":"5eb89c0a562c1bb1f8a0fa57e6d738246804e9cbddfc28eb612fa8a9a9376b38"} Oct 30 05:47:24.429595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.429579 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" event={"ID":"dcf3fb25-226b-41ab-9b75-b0bf81a864f1","Type":"ContainerStarted","Data":"eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c"} Oct 30 05:47:24.519873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.519794 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" podStartSLOduration=2.5197714060000003 podStartE2EDuration="2.519771406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:24.484500685 +0000 UTC m=+902.924877879" watchObservedRunningTime="2026-10-30 05:47:24.519771406 +0000 UTC m=+902.960148600" Oct 30 05:47:24.535893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:24.535815 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" podStartSLOduration=2.535793866 podStartE2EDuration="2.535793866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:24.533382765 +0000 UTC m=+902.973759960" watchObservedRunningTime="2026-10-30 05:47:24.535793866 +0000 UTC m=+902.976171062" Oct 30 05:47:25.321630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.321588 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:25.321823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.321654 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:25.438606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.438572 6171 generic.go:356] "Generic (PLEG): container finished" podID="dcf3fb25-226b-41ab-9b75-b0bf81a864f1" containerID="5eb89c0a562c1bb1f8a0fa57e6d738246804e9cbddfc28eb612fa8a9a9376b38" exitCode=0 Oct 30 05:47:25.439113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.438659 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" event={"ID":"dcf3fb25-226b-41ab-9b75-b0bf81a864f1","Type":"ContainerDied","Data":"5eb89c0a562c1bb1f8a0fa57e6d738246804e9cbddfc28eb612fa8a9a9376b38"} Oct 30 05:47:25.440452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.440425 6171 generic.go:356] "Generic (PLEG): container finished" podID="5751de18-3728-4198-a097-d149fbddbc61" containerID="b174338d459b16d246019f87f7c18d8c87e2700bb0fe0e1fd43bc01cfcc5d392" exitCode=0 Oct 30 05:47:25.440579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:25.440505 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" event={"ID":"5751de18-3728-4198-a097-d149fbddbc61","Type":"ContainerDied","Data":"b174338d459b16d246019f87f7c18d8c87e2700bb0fe0e1fd43bc01cfcc5d392"} Oct 30 05:47:26.110168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.110132 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8"] Oct 30 05:47:26.110747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.110732 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8eb26885-8ce8-4af2-8736-606701a15d7e" containerName="revoke-keys" Oct 30 05:47:26.110790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.110749 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8eb26885-8ce8-4af2-8736-606701a15d7e" containerName="revoke-keys" Oct 30 05:47:26.110869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.110861 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8eb26885-8ce8-4af2-8736-606701a15d7e" containerName="revoke-keys" Oct 30 05:47:26.114408 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.114391 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.137629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.137593 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8"] Oct 30 05:47:26.222582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.222544 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.222582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.222596 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.222881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.222679 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnknq\" (UniqueName: \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.320966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.320931 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:26.321192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.320982 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:26.323680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.323653 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.323730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.323702 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.323765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.323731 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnknq\" (UniqueName: \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.324350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.324327 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.326290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.326267 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.350311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.350277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnknq\" (UniqueName: \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\") pod \"maas-api-revoke-sub-4c8328ad-4sdr8\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.405980 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:47:26.405875 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:47:26.428292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.428262 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:26.756023 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.755994 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:26.758843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.758819 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:26.810308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.808390 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8"] Oct 30 05:47:26.813722 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:26.813688 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13296f31_2f34_4cb2_83f6_f6bf3c54bf4d.slice/crio-a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef WatchSource:0}: Error finding container a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef: Status 404 returned error can't find the container with id a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef Oct 30 05:47:26.832511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832463 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\" (UniqueName: \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\") pod \"5751de18-3728-4198-a097-d149fbddbc61\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " Oct 30 05:47:26.832690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832544 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\") pod \"5751de18-3728-4198-a097-d149fbddbc61\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " Oct 30 05:47:26.832690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832607 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\") pod \"5751de18-3728-4198-a097-d149fbddbc61\" (UID: \"5751de18-3728-4198-a097-d149fbddbc61\") " Oct 30 05:47:26.832690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832643 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\" (UniqueName: \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\") pod \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " Oct 30 05:47:26.832893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832717 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\") pod \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " Oct 30 05:47:26.832893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.832756 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\") pod \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\" (UID: \"dcf3fb25-226b-41ab-9b75-b0bf81a864f1\") " Oct 30 05:47:26.833320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.833200 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca" pod "5751de18-3728-4198-a097-d149fbddbc61" (UID: "5751de18-3728-4198-a097-d149fbddbc61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:26.834394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.834366 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5751de18-3728-4198-a097-d149fbddbc61-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:26.834727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.834700 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca" pod "dcf3fb25-226b-41ab-9b75-b0bf81a864f1" (UID: "dcf3fb25-226b-41ab-9b75-b0bf81a864f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:26.835366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.835324 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh" pod "5751de18-3728-4198-a097-d149fbddbc61" (UID: "5751de18-3728-4198-a097-d149fbddbc61"). InnerVolumeSpecName "kube-api-access-l7brh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:26.835531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.835500 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca" pod "5751de18-3728-4198-a097-d149fbddbc61" (UID: "5751de18-3728-4198-a097-d149fbddbc61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:26.837112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.837025 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn" pod "dcf3fb25-226b-41ab-9b75-b0bf81a864f1" (UID: "dcf3fb25-226b-41ab-9b75-b0bf81a864f1"). InnerVolumeSpecName "kube-api-access-sctmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:26.837220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.837157 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca" pod "dcf3fb25-226b-41ab-9b75-b0bf81a864f1" (UID: "dcf3fb25-226b-41ab-9b75-b0bf81a864f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:26.935801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.935753 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7brh\" (UniqueName: \"kubernetes.io/projected/5751de18-3728-4198-a097-d149fbddbc61-kube-api-access-l7brh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:26.935801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.935798 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5751de18-3728-4198-a097-d149fbddbc61-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:26.936021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.935813 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sctmn\" (UniqueName: \"kubernetes.io/projected/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-kube-api-access-sctmn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:26.936021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.935829 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:26.936021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:26.935844 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcf3fb25-226b-41ab-9b75-b0bf81a864f1-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:27.320673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.320574 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:27.320673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.320636 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" Oct 30 05:47:27.458346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.458304 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" event={"ID":"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d","Type":"ContainerStarted","Data":"0e9d15d111f7befff0a7ab66cbb37af4878f48d15e3ae791749a846b68b5955a"} Oct 30 05:47:27.458602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.458374 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" event={"ID":"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d","Type":"ContainerStarted","Data":"a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef"} Oct 30 05:47:27.459968 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.459938 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" Oct 30 05:47:27.460132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.459972 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5" event={"ID":"5751de18-3728-4198-a097-d149fbddbc61","Type":"ContainerDied","Data":"0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3"} Oct 30 05:47:27.460132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.460033 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eb5c137a00435ed5069c0d9ab4a4cdeee880d8c358bda1245a00a05f3f958d3" Oct 30 05:47:27.461528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.461503 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" Oct 30 05:47:27.461635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.461520 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7" event={"ID":"dcf3fb25-226b-41ab-9b75-b0bf81a864f1","Type":"ContainerDied","Data":"eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c"} Oct 30 05:47:27.461635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.461568 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb645ce0b1024a6a10dd19eeaefb8ebbea3d86ef4840ac34167db5ad9aae011c" Oct 30 05:47:27.504315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:27.504254 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" podStartSLOduration=1.504238293 podStartE2EDuration="1.504238293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:27.503165542 +0000 UTC m=+905.943542734" watchObservedRunningTime="2026-10-30 05:47:27.504238293 +0000 UTC m=+905.944615486" Oct 30 05:47:28.478018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:28.477966 6171 generic.go:356] "Generic (PLEG): container finished" podID="13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" containerID="0e9d15d111f7befff0a7ab66cbb37af4878f48d15e3ae791749a846b68b5955a" exitCode=0 Oct 30 05:47:28.478533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:28.478067 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" event={"ID":"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d","Type":"ContainerDied","Data":"0e9d15d111f7befff0a7ab66cbb37af4878f48d15e3ae791749a846b68b5955a"} Oct 30 05:47:28.647690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:28.647658 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:28.648107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:28.648082 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:28.649484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:28.649462 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:47:29.630915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.630885 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:29.674891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.674858 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\" (UniqueName: \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\") pod \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " Oct 30 05:47:29.675095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.675069 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\") pod \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " Oct 30 05:47:29.675159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.675116 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\") pod \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\" (UID: \"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d\") " Oct 30 05:47:29.676285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.676252 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca" pod "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" (UID: "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:29.678385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.678343 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca" pod "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" (UID: "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:29.678536 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.678503 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq" pod "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" (UID: "13296f31-2f34-4cb2-83f6-f6bf3c54bf4d"). InnerVolumeSpecName "kube-api-access-nnknq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:29.777074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.776966 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:29.777074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.776999 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:29.777074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:29.777010 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnknq\" (UniqueName: \"kubernetes.io/projected/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d-kube-api-access-nnknq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:30.492947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:30.492911 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" Oct 30 05:47:30.492947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:30.492935 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8" event={"ID":"13296f31-2f34-4cb2-83f6-f6bf3c54bf4d","Type":"ContainerDied","Data":"a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef"} Oct 30 05:47:30.493201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:30.492988 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7d449125f36cd88d9418c81efe5e6e3b7d97149846bdc97e922da99ac2db7ef" Oct 30 05:47:31.544711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.544610 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x"] Oct 30 05:47:31.545289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545271 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5751de18-3728-4198-a097-d149fbddbc61" containerName="revoke-keys" Oct 30 05:47:31.545332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545291 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5751de18-3728-4198-a097-d149fbddbc61" containerName="revoke-keys" Oct 30 05:47:31.545332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545306 6171 cpu_manager.go:414] "containerMap: removing container" podUID="13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" containerName="revoke-keys" Oct 30 05:47:31.545332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545312 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" containerName="revoke-keys" Oct 30 05:47:31.545332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545321 6171 cpu_manager.go:414] "containerMap: removing container" podUID="dcf3fb25-226b-41ab-9b75-b0bf81a864f1" containerName="revoke-keys" Oct 30 05:47:31.545332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545325 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcf3fb25-226b-41ab-9b75-b0bf81a864f1" containerName="revoke-keys" Oct 30 05:47:31.545482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545450 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5751de18-3728-4198-a097-d149fbddbc61" containerName="revoke-keys" Oct 30 05:47:31.545482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545459 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" containerName="revoke-keys" Oct 30 05:47:31.545482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.545465 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcf3fb25-226b-41ab-9b75-b0bf81a864f1" containerName="revoke-keys" Oct 30 05:47:31.551071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.550713 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.573645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.573616 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x"] Oct 30 05:47:31.581445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.581415 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:31.581653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.581463 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:31.581753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.581530 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:47:31.581753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.581588 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:31.599836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.599790 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48w86\" (UniqueName: \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.600010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.599885 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.600199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.600173 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.717113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.706952 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.717113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.707251 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48w86\" (UniqueName: \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.717113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.707319 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.717113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.708920 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.717113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.712555 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.739508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.739469 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48w86\" (UniqueName: \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\") pod \"maas-api-revoke-sub-7d8586d8-fc75x\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:31.868434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:31.868329 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:32.179502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:32.179470 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x"] Oct 30 05:47:32.179967 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:32.179944 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefd1f3e6_7a65_4c97_a924_fc296c385ef5.slice/crio-03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517 WatchSource:0}: Error finding container 03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517: Status 404 returned error can't find the container with id 03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517 Oct 30 05:47:32.508474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:32.508439 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" event={"ID":"efd1f3e6-7a65-4c97-a924-fc296c385ef5","Type":"ContainerStarted","Data":"22017ca6aebaffda1e9dafc352eea022adc4524394c5a3896cb8fb225c99628a"} Oct 30 05:47:32.508598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:32.508499 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" event={"ID":"efd1f3e6-7a65-4c97-a924-fc296c385ef5","Type":"ContainerStarted","Data":"03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517"} Oct 30 05:47:32.569003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:32.568943 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" podStartSLOduration=1.568929105 podStartE2EDuration="1.568929105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:32.566199298 +0000 UTC m=+911.006576490" watchObservedRunningTime="2026-10-30 05:47:32.568929105 +0000 UTC m=+911.009306298" Oct 30 05:47:33.037710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.037672 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh"] Oct 30 05:47:33.042289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.042265 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.071684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.071656 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh"] Oct 30 05:47:33.124949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.124918 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.125132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.125100 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmmft\" (UniqueName: \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.125262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.125206 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.226829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.226786 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.227091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.226883 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmmft\" (UniqueName: \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.227091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.226927 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.227556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.227527 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.229479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.229460 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.251902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.251861 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmmft\" (UniqueName: \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\") pod \"maas-api-revoke-sub-a6ec72c1-57zxh\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.357910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.357817 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:33.517128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.517089 6171 generic.go:356] "Generic (PLEG): container finished" podID="efd1f3e6-7a65-4c97-a924-fc296c385ef5" containerID="22017ca6aebaffda1e9dafc352eea022adc4524394c5a3896cb8fb225c99628a" exitCode=0 Oct 30 05:47:33.517318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.517160 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" event={"ID":"efd1f3e6-7a65-4c97-a924-fc296c385ef5","Type":"ContainerDied","Data":"22017ca6aebaffda1e9dafc352eea022adc4524394c5a3896cb8fb225c99628a"} Oct 30 05:47:33.704037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:33.704013 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh"] Oct 30 05:47:34.526332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.526299 6171 generic.go:356] "Generic (PLEG): container finished" podID="5016c6bb-9b58-45e1-885f-cee9ddf915d7" containerID="832fabdf50f69afbd9c2cecc26adca4eed3ebaf3eec2058b1511e644bdb14b3b" exitCode=0 Oct 30 05:47:34.526510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.526385 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" event={"ID":"5016c6bb-9b58-45e1-885f-cee9ddf915d7","Type":"ContainerDied","Data":"832fabdf50f69afbd9c2cecc26adca4eed3ebaf3eec2058b1511e644bdb14b3b"} Oct 30 05:47:34.526510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.526468 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" event={"ID":"5016c6bb-9b58-45e1-885f-cee9ddf915d7","Type":"ContainerStarted","Data":"f4f40a742d4a078afde96a3d19fef063a2ad4953d3368959e263135f537fa6ea"} Oct 30 05:47:34.688020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.687992 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:34.747374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.747344 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\" (UniqueName: \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\") pod \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " Oct 30 05:47:34.747878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.747513 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\") pod \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " Oct 30 05:47:34.747878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.747539 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\") pod \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\" (UID: \"efd1f3e6-7a65-4c97-a924-fc296c385ef5\") " Oct 30 05:47:34.748004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.747891 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca" pod "efd1f3e6-7a65-4c97-a924-fc296c385ef5" (UID: "efd1f3e6-7a65-4c97-a924-fc296c385ef5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:34.748189 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.748090 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:34.749875 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.749849 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86" pod "efd1f3e6-7a65-4c97-a924-fc296c385ef5" (UID: "efd1f3e6-7a65-4c97-a924-fc296c385ef5"). InnerVolumeSpecName "kube-api-access-48w86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:34.749875 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.749861 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca" pod "efd1f3e6-7a65-4c97-a924-fc296c385ef5" (UID: "efd1f3e6-7a65-4c97-a924-fc296c385ef5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:34.849672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.849633 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48w86\" (UniqueName: \"kubernetes.io/projected/efd1f3e6-7a65-4c97-a924-fc296c385ef5-kube-api-access-48w86\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:34.849672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:34.849663 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efd1f3e6-7a65-4c97-a924-fc296c385ef5-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:35.535719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.535680 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" event={"ID":"efd1f3e6-7a65-4c97-a924-fc296c385ef5","Type":"ContainerDied","Data":"03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517"} Oct 30 05:47:35.536448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.535743 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03e2e0637465f044aded618254b33726a604ca740d74b301addc51a5608bc517" Oct 30 05:47:35.536448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.535654 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x" Oct 30 05:47:35.558452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.558422 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd"] Oct 30 05:47:35.559084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.559061 6171 cpu_manager.go:414] "containerMap: removing container" podUID="efd1f3e6-7a65-4c97-a924-fc296c385ef5" containerName="revoke-keys" Oct 30 05:47:35.559084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.559082 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efd1f3e6-7a65-4c97-a924-fc296c385ef5" containerName="revoke-keys" Oct 30 05:47:35.559258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.559248 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="efd1f3e6-7a65-4c97-a924-fc296c385ef5" containerName="revoke-keys" Oct 30 05:47:35.562641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.562625 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.584375 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.584346 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd"] Oct 30 05:47:35.672076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.661744 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmlb5\" (UniqueName: \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.672076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.661879 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.727065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.661936 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.761822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.761794 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:35.827376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.827270 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\" (UniqueName: \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\") pod \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " Oct 30 05:47:35.827578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.827421 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\") pod \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " Oct 30 05:47:35.827578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.827463 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\") pod \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\" (UID: \"5016c6bb-9b58-45e1-885f-cee9ddf915d7\") " Oct 30 05:47:35.827973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.827941 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmlb5\" (UniqueName: \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.828128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.828097 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.828128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.828110 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca" pod "5016c6bb-9b58-45e1-885f-cee9ddf915d7" (UID: "5016c6bb-9b58-45e1-885f-cee9ddf915d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:35.828269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.828175 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.828826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.828755 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:35.829105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.829000 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.832832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.832799 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca" pod "5016c6bb-9b58-45e1-885f-cee9ddf915d7" (UID: "5016c6bb-9b58-45e1-885f-cee9ddf915d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:35.832832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.832821 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft" pod "5016c6bb-9b58-45e1-885f-cee9ddf915d7" (UID: "5016c6bb-9b58-45e1-885f-cee9ddf915d7"). InnerVolumeSpecName "kube-api-access-cmmft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:35.833215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.833173 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.852498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.852469 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmlb5\" (UniqueName: \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\") pod \"maas-api-revoke-sub-9d1e101f-dsvhd\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.878059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.878013 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:35.908510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.908479 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 05:47:35.909357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.909299 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5016c6bb-9b58-45e1-885f-cee9ddf915d7" containerName="revoke-keys" Oct 30 05:47:35.909357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.909321 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5016c6bb-9b58-45e1-885f-cee9ddf915d7" containerName="revoke-keys" Oct 30 05:47:35.909604 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.909472 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5016c6bb-9b58-45e1-885f-cee9ddf915d7" containerName="revoke-keys" Oct 30 05:47:35.913437 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.913409 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:35.929153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.929119 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-1354a5-openshift-default-dockercfg-8c2kz\"" Oct 30 05:47:35.929755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.929740 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmmft\" (UniqueName: \"kubernetes.io/projected/5016c6bb-9b58-45e1-885f-cee9ddf915d7-kube-api-access-cmmft\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:35.929805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.929760 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5016c6bb-9b58-45e1-885f-cee9ddf915d7-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:35.939423 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:35.939389 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 05:47:36.030677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.030619 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.030677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.030676 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.030927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.030792 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.030927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.030883 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.031007 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.030958 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jl6vd\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.031140 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.031123 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.031226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.031212 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.031292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.031281 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.031329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.031303 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132394 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132470 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132492 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132516 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132536 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132562 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132600 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jl6vd\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.132794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132739 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.133160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132992 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.133160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.132998 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.133160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.133079 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.133314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.133229 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.133574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.133508 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.135157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.135137 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.135269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.135248 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.157584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.157554 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.157736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.157625 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jl6vd\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.199329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.199299 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd"] Oct 30 05:47:36.200110 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:36.200063 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e5ab780_b600_45e4_8806_d954c4f17e6e.slice/crio-22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d WatchSource:0}: Error finding container 22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d: Status 404 returned error can't find the container with id 22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d Oct 30 05:47:36.229738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.229708 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:36.503725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.503694 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t"] Oct 30 05:47:36.507926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.507903 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.528129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.528100 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t"] Oct 30 05:47:36.538581 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.538551 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.538786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.538608 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.539059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.538860 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxz8c\" (UniqueName: \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.546117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.546080 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" event={"ID":"1e5ab780-b600-45e4-8806-d954c4f17e6e","Type":"ContainerStarted","Data":"71b9a77a806ae2d22faf7558bb7a0a5f967205e962a11edfbffad47120bc141f"} Oct 30 05:47:36.546294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.546154 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" event={"ID":"1e5ab780-b600-45e4-8806-d954c4f17e6e","Type":"ContainerStarted","Data":"22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d"} Oct 30 05:47:36.547934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.547898 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" Oct 30 05:47:36.547934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.547922 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh" event={"ID":"5016c6bb-9b58-45e1-885f-cee9ddf915d7","Type":"ContainerDied","Data":"f4f40a742d4a078afde96a3d19fef063a2ad4953d3368959e263135f537fa6ea"} Oct 30 05:47:36.548207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.547974 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4f40a742d4a078afde96a3d19fef063a2ad4953d3368959e263135f537fa6ea" Oct 30 05:47:36.574234 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:36.562877 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ff11b1_4490_4b67_b1fe_2b319f67f239.slice/crio-4dd2f3f870469b9bf2d3a5ce00d30d1ae1d55ec2c871f24297b54cb40de0e70e WatchSource:0}: Error finding container 4dd2f3f870469b9bf2d3a5ce00d30d1ae1d55ec2c871f24297b54cb40de0e70e: Status 404 returned error can't find the container with id 4dd2f3f870469b9bf2d3a5ce00d30d1ae1d55ec2c871f24297b54cb40de0e70e Oct 30 05:47:36.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.565660 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:36.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.565735 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:36.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.565770 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:47:36.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.565737 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 05:47:36.596311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.596259 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" podStartSLOduration=1.596243978 podStartE2EDuration="1.596243978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:36.594687641 +0000 UTC m=+915.035064850" watchObservedRunningTime="2026-10-30 05:47:36.596243978 +0000 UTC m=+915.036621206" Oct 30 05:47:36.640808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.640740 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxz8c\" (UniqueName: \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.640974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.640836 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.640974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.640884 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.641954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.641932 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.643619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.643593 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.667805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.667767 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxz8c\" (UniqueName: \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\") pod \"maas-api-revoke-sub-6a49ea64-cmt6t\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:36.823836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:36.823751 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:37.139132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.139105 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t"] Oct 30 05:47:37.155098 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:37.155065 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5253fdb2_30fb_4116_b9e8_20220b6a06cf.slice/crio-a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0 WatchSource:0}: Error finding container a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0: Status 404 returned error can't find the container with id a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0 Oct 30 05:47:37.557660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.557236 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" event={"ID":"79ff11b1-4490-4b67-b1fe-2b319f67f239","Type":"ContainerStarted","Data":"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98"} Oct 30 05:47:37.557660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.557305 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" event={"ID":"79ff11b1-4490-4b67-b1fe-2b319f67f239","Type":"ContainerStarted","Data":"4dd2f3f870469b9bf2d3a5ce00d30d1ae1d55ec2c871f24297b54cb40de0e70e"} Oct 30 05:47:37.559232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.559201 6171 generic.go:356] "Generic (PLEG): container finished" podID="1e5ab780-b600-45e4-8806-d954c4f17e6e" containerID="71b9a77a806ae2d22faf7558bb7a0a5f967205e962a11edfbffad47120bc141f" exitCode=0 Oct 30 05:47:37.559364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.559282 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" event={"ID":"1e5ab780-b600-45e4-8806-d954c4f17e6e","Type":"ContainerDied","Data":"71b9a77a806ae2d22faf7558bb7a0a5f967205e962a11edfbffad47120bc141f"} Oct 30 05:47:37.560712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.560684 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" event={"ID":"5253fdb2-30fb-4116-b9e8-20220b6a06cf","Type":"ContainerStarted","Data":"a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0"} Oct 30 05:47:37.608399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.608311 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podStartSLOduration=2.608292863 podStartE2EDuration="2.608292863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:37.605562386 +0000 UTC m=+916.045939580" watchObservedRunningTime="2026-10-30 05:47:37.608292863 +0000 UTC m=+916.048670056" Oct 30 05:47:37.621583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.621551 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg"] Oct 30 05:47:37.625849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.625828 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.638992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.638951 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg"] Oct 30 05:47:37.758700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.758660 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5mqd\" (UniqueName: \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.758927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.758798 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.758995 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.758950 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.761930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.761904 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:47:37.859762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.859723 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.860324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.859955 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.860324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.860195 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5mqd\" (UniqueName: \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.860685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.860666 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.862573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.862546 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.908243 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.908208 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5mqd\" (UniqueName: \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\") pod \"maas-api-revoke-sub-7e81b3e0-m65cg\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:37.943801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:37.943766 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:38.230224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.230191 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:38.231841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.231813 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:38.231998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.231873 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:38.345669 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.345612 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg"] Oct 30 05:47:38.642528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.642019 6171 generic.go:356] "Generic (PLEG): container finished" podID="5253fdb2-30fb-4116-b9e8-20220b6a06cf" containerID="ec2fbff575b272983824f00133c5744a6703b7f2ad3566b62034cf5664775157" exitCode=0 Oct 30 05:47:38.642528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.642134 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" event={"ID":"5253fdb2-30fb-4116-b9e8-20220b6a06cf","Type":"ContainerDied","Data":"ec2fbff575b272983824f00133c5744a6703b7f2ad3566b62034cf5664775157"} Oct 30 05:47:38.645855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:38.644902 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" event={"ID":"7086963e-3b9c-4cf5-a783-6a3f30e306e2","Type":"ContainerStarted","Data":"ecc3615936c74da755effb7f35b9350bfcd3579eacd891e00a1566e42fbf8165"} Oct 30 05:47:39.259087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.257867 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:39.259087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.257929 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:39.337609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.337247 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:39.448160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.435855 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\" (UniqueName: \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\") pod \"1e5ab780-b600-45e4-8806-d954c4f17e6e\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " Oct 30 05:47:39.448160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.435968 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\") pod \"1e5ab780-b600-45e4-8806-d954c4f17e6e\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " Oct 30 05:47:39.448160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.436017 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\") pod \"1e5ab780-b600-45e4-8806-d954c4f17e6e\" (UID: \"1e5ab780-b600-45e4-8806-d954c4f17e6e\") " Oct 30 05:47:39.448160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.448017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca" pod "1e5ab780-b600-45e4-8806-d954c4f17e6e" (UID: "1e5ab780-b600-45e4-8806-d954c4f17e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:39.463668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.463599 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5" pod "1e5ab780-b600-45e4-8806-d954c4f17e6e" (UID: "1e5ab780-b600-45e4-8806-d954c4f17e6e"). InnerVolumeSpecName "kube-api-access-zmlb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:39.481267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.481194 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca" pod "1e5ab780-b600-45e4-8806-d954c4f17e6e" (UID: "1e5ab780-b600-45e4-8806-d954c4f17e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:39.567131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.542001 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmlb5\" (UniqueName: \"kubernetes.io/projected/1e5ab780-b600-45e4-8806-d954c4f17e6e-kube-api-access-zmlb5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:39.567131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.542062 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:39.567131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:39.542080 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e5ab780-b600-45e4-8806-d954c4f17e6e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:40.127868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.127762 6171 generic.go:356] "Generic (PLEG): container finished" podID="7086963e-3b9c-4cf5-a783-6a3f30e306e2" containerID="be093b6f847802eb6335d03e3c303b0b16a81900533e911330f86e1812c9e642" exitCode=0 Oct 30 05:47:40.128522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.128489 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" event={"ID":"7086963e-3b9c-4cf5-a783-6a3f30e306e2","Type":"ContainerDied","Data":"be093b6f847802eb6335d03e3c303b0b16a81900533e911330f86e1812c9e642"} Oct 30 05:47:40.156086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.152298 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" Oct 30 05:47:40.168676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.164772 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd" event={"ID":"1e5ab780-b600-45e4-8806-d954c4f17e6e","Type":"ContainerDied","Data":"22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d"} Oct 30 05:47:40.168676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.164855 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22bb6dbdc9eab093bdcde560e23948aee1ee6a468e881baea87a13928acd155d" Oct 30 05:47:40.230646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.230568 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:40.231034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.230960 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:40.274576 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:47:40.274533 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7086963e_3b9c_4cf5_a783_6a3f30e306e2.slice/crio-conmon-be093b6f847802eb6335d03e3c303b0b16a81900533e911330f86e1812c9e642.scope\": RecentStats: unable to find data in memory cache]" Oct 30 05:47:40.765335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.764913 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:40.871086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.861834 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\") pod \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " Oct 30 05:47:40.871086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.862130 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\") pod \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " Oct 30 05:47:40.871086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.862172 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\" (UniqueName: \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\") pod \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\" (UID: \"5253fdb2-30fb-4116-b9e8-20220b6a06cf\") " Oct 30 05:47:40.875712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.875308 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca" pod "5253fdb2-30fb-4116-b9e8-20220b6a06cf" (UID: "5253fdb2-30fb-4116-b9e8-20220b6a06cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:40.912770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.912724 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c" pod "5253fdb2-30fb-4116-b9e8-20220b6a06cf" (UID: "5253fdb2-30fb-4116-b9e8-20220b6a06cf"). InnerVolumeSpecName "kube-api-access-pxz8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:40.952337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.952264 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca" pod "5253fdb2-30fb-4116-b9e8-20220b6a06cf" (UID: "5253fdb2-30fb-4116-b9e8-20220b6a06cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:40.990074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.980021 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:40.990074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.980092 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxz8c\" (UniqueName: \"kubernetes.io/projected/5253fdb2-30fb-4116-b9e8-20220b6a06cf-kube-api-access-pxz8c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:40.990074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:40.980110 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5253fdb2-30fb-4116-b9e8-20220b6a06cf-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:41.234531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.231789 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:41.234531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.231853 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:41.251081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.248573 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" Oct 30 05:47:41.251453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.251341 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t" event={"ID":"5253fdb2-30fb-4116-b9e8-20220b6a06cf","Type":"ContainerDied","Data":"a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0"} Oct 30 05:47:41.251453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.251413 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5abbf3ff5ee383dad72f6357e97fe10a0e3cd6ddb81a3de0025b2923d8738a0" Oct 30 05:47:41.908035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:41.907986 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.026136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.008175 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.026891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.026835 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.140221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.140178 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:47:42.141005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.140500 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.214537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.214503 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:42.245073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.244686 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:42.249069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.245280 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:42.424140 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.423184 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:47:42.424140 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.423258 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.478072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.477435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\") pod \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " Oct 30 05:47:42.478072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.477586 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\") pod \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " Oct 30 05:47:42.478072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.477646 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\" (UniqueName: \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\") pod \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\" (UID: \"7086963e-3b9c-4cf5-a783-6a3f30e306e2\") " Oct 30 05:47:42.503073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.480007 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca" pod "7086963e-3b9c-4cf5-a783-6a3f30e306e2" (UID: "7086963e-3b9c-4cf5-a783-6a3f30e306e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:42.554627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.554090 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" event={"ID":"7086963e-3b9c-4cf5-a783-6a3f30e306e2","Type":"ContainerDied","Data":"ecc3615936c74da755effb7f35b9350bfcd3579eacd891e00a1566e42fbf8165"} Oct 30 05:47:42.554627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.554169 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ecc3615936c74da755effb7f35b9350bfcd3579eacd891e00a1566e42fbf8165" Oct 30 05:47:42.554627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.554359 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg" Oct 30 05:47:42.577114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.576269 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd" pod "7086963e-3b9c-4cf5-a783-6a3f30e306e2" (UID: "7086963e-3b9c-4cf5-a783-6a3f30e306e2"). InnerVolumeSpecName "kube-api-access-c5mqd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:42.577114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.576373 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca" pod "7086963e-3b9c-4cf5-a783-6a3f30e306e2" (UID: "7086963e-3b9c-4cf5-a783-6a3f30e306e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:42.579666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.579575 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:42.579666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.579610 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7086963e-3b9c-4cf5-a783-6a3f30e306e2-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:42.579666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.579626 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5mqd\" (UniqueName: \"kubernetes.io/projected/7086963e-3b9c-4cf5-a783-6a3f30e306e2-kube-api-access-c5mqd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:42.853924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.853865 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:42.870538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:42.867172 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:47:43.230770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:43.230738 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:43.231246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:43.230795 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:44.230390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:44.230346 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 05:47:44.230599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:44.230424 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 05:47:45.938648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:45.938602 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:45.939256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:45.939234 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:45.964924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:45.964892 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 05:47:48.099652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.099617 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7"] Oct 30 05:47:48.100218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100204 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1e5ab780-b600-45e4-8806-d954c4f17e6e" containerName="revoke-keys" Oct 30 05:47:48.100277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100219 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e5ab780-b600-45e4-8806-d954c4f17e6e" containerName="revoke-keys" Oct 30 05:47:48.100277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100242 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5253fdb2-30fb-4116-b9e8-20220b6a06cf" containerName="revoke-keys" Oct 30 05:47:48.100277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100247 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5253fdb2-30fb-4116-b9e8-20220b6a06cf" containerName="revoke-keys" Oct 30 05:47:48.100380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100282 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7086963e-3b9c-4cf5-a783-6a3f30e306e2" containerName="revoke-keys" Oct 30 05:47:48.100380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100289 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7086963e-3b9c-4cf5-a783-6a3f30e306e2" containerName="revoke-keys" Oct 30 05:47:48.100380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100368 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7086963e-3b9c-4cf5-a783-6a3f30e306e2" containerName="revoke-keys" Oct 30 05:47:48.100380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100376 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e5ab780-b600-45e4-8806-d954c4f17e6e" containerName="revoke-keys" Oct 30 05:47:48.100521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.100383 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5253fdb2-30fb-4116-b9e8-20220b6a06cf" containerName="revoke-keys" Oct 30 05:47:48.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.104705 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.126057 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.126014 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7"] Oct 30 05:47:48.132386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.132356 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:48.132541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.132449 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:48.132541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.132453 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:48.132664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.132586 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:47:48.281859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.281825 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.282125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.281994 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.282212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.282167 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxw6\" (UniqueName: \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.383651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.383559 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxw6\" (UniqueName: \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.383651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.383647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.383853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.383756 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.384400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.384381 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.386260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.386243 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.409725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.409693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxw6\" (UniqueName: \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\") pod \"maas-api-revoke-sub-3021f688-c24k7\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.419609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.419583 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:48.738965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:48.738931 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7"] Oct 30 05:47:48.739116 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:47:48.739007 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43634ed9_047e_4f71_abad_d76717c3a8e9.slice/crio-dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0 WatchSource:0}: Error finding container dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0: Status 404 returned error can't find the container with id dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0 Oct 30 05:47:49.618199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:49.618161 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" event={"ID":"43634ed9-047e-4f71-abad-d76717c3a8e9","Type":"ContainerStarted","Data":"18c00e0ff403506f1c8567c17677b818edf0cc0a3669e8048f7cfe6159cede68"} Oct 30 05:47:49.618711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:49.618214 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" event={"ID":"43634ed9-047e-4f71-abad-d76717c3a8e9","Type":"ContainerStarted","Data":"dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0"} Oct 30 05:47:49.660614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:49.660561 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" podStartSLOduration=1.660542672 podStartE2EDuration="1.660542672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:49.659431001 +0000 UTC m=+928.099808207" watchObservedRunningTime="2026-10-30 05:47:49.660542672 +0000 UTC m=+928.100919865" Oct 30 05:47:50.628794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:50.628755 6171 generic.go:356] "Generic (PLEG): container finished" podID="43634ed9-047e-4f71-abad-d76717c3a8e9" containerID="18c00e0ff403506f1c8567c17677b818edf0cc0a3669e8048f7cfe6159cede68" exitCode=0 Oct 30 05:47:50.629223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:50.628830 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" event={"ID":"43634ed9-047e-4f71-abad-d76717c3a8e9","Type":"ContainerDied","Data":"18c00e0ff403506f1c8567c17677b818edf0cc0a3669e8048f7cfe6159cede68"} Oct 30 05:47:51.799109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.799085 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:51.827374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.827336 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\") pod \"43634ed9-047e-4f71-abad-d76717c3a8e9\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " Oct 30 05:47:51.827580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.827399 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\") pod \"43634ed9-047e-4f71-abad-d76717c3a8e9\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " Oct 30 05:47:51.827580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.827441 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\" (UniqueName: \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\") pod \"43634ed9-047e-4f71-abad-d76717c3a8e9\" (UID: \"43634ed9-047e-4f71-abad-d76717c3a8e9\") " Oct 30 05:47:51.827776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.827750 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca" pod "43634ed9-047e-4f71-abad-d76717c3a8e9" (UID: "43634ed9-047e-4f71-abad-d76717c3a8e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:51.829770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.829740 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6" pod "43634ed9-047e-4f71-abad-d76717c3a8e9" (UID: "43634ed9-047e-4f71-abad-d76717c3a8e9"). InnerVolumeSpecName "kube-api-access-tpxw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:51.829872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.829775 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca" pod "43634ed9-047e-4f71-abad-d76717c3a8e9" (UID: "43634ed9-047e-4f71-abad-d76717c3a8e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:51.929029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.928940 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:51.929029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.928974 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43634ed9-047e-4f71-abad-d76717c3a8e9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:51.929029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:51.928986 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxw6\" (UniqueName: \"kubernetes.io/projected/43634ed9-047e-4f71-abad-d76717c3a8e9-kube-api-access-tpxw6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:47:52.644873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:52.644847 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" Oct 30 05:47:52.645089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:52.644884 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7" event={"ID":"43634ed9-047e-4f71-abad-d76717c3a8e9","Type":"ContainerDied","Data":"dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0"} Oct 30 05:47:52.645089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:52.644958 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dce39914c2165dde32754f1ace7cfc06f6e936a6b8a1024068870fffabc159c0" Oct 30 05:47:56.401065 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:47:56.400999 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:47:57.076927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:57.076878 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:47:57.077186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:47:57.076969 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:02.077734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.077698 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 05:48:02.078386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.078370 6171 cpu_manager.go:414] "containerMap: removing container" podUID="43634ed9-047e-4f71-abad-d76717c3a8e9" containerName="revoke-keys" Oct 30 05:48:02.078449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.078388 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43634ed9-047e-4f71-abad-d76717c3a8e9" containerName="revoke-keys" Oct 30 05:48:02.078493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.078483 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="43634ed9-047e-4f71-abad-d76717c3a8e9" containerName="revoke-keys" Oct 30 05:48:02.095801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.095768 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.133901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.133867 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 05:48:02.144271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144220 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144511 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144702 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144775 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.144844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144825 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2ztc\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.145004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.144980 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.145184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.145163 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.246780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246732 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.246780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246788 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246815 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246845 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246874 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246938 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.246981 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2ztc\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247103 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247203 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247604 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247652 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247637 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247684 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.247971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.247949 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.249588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.249566 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.249676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.249617 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.271033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.270992 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.271214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.271129 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2ztc\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\") pod \"maas-default-gateway-openshift-default-58847f9fc5-2c9x4\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.451638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.451546 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:02.780786 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:48:02.780543 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51093a2f_5b98_4388_b1ba_834f8b5c4b1f.slice/crio-1d3e1aca8fddfc51fa6b5023113a22c3789229bc775e5f6eeb7e639a1224cfbb WatchSource:0}: Error finding container 1d3e1aca8fddfc51fa6b5023113a22c3789229bc775e5f6eeb7e639a1224cfbb: Status 404 returned error can't find the container with id 1d3e1aca8fddfc51fa6b5023113a22c3789229bc775e5f6eeb7e639a1224cfbb Oct 30 05:48:02.780936 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.780871 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 05:48:02.783369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.783334 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:02.783513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.783416 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:02.783513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:02.783462 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:03.776301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:03.776261 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" event={"ID":"51093a2f-5b98-4388-b1ba-834f8b5c4b1f","Type":"ContainerStarted","Data":"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219"} Oct 30 05:48:03.776711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:03.776327 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" event={"ID":"51093a2f-5b98-4388-b1ba-834f8b5c4b1f","Type":"ContainerStarted","Data":"1d3e1aca8fddfc51fa6b5023113a22c3789229bc775e5f6eeb7e639a1224cfbb"} Oct 30 05:48:03.827631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:03.827460 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podStartSLOduration=1.827441244 podStartE2EDuration="1.827441244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:03.826679829 +0000 UTC m=+942.267057023" watchObservedRunningTime="2026-10-30 05:48:03.827441244 +0000 UTC m=+942.267818438" Oct 30 05:48:04.452383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:04.452342 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:04.453924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:04.453898 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:04.453990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:04.453958 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:05.452390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:05.452351 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:05.452819 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:05.452410 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:06.452709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:06.452671 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:06.453171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:06.452735 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:07.452807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:07.452768 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:07.453459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:07.452830 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:08.099073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.080698 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:48:08.452406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.452313 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:08.452582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.452398 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:08.657219 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.657181 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 05:48:08.663245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.663210 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.694070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.689896 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-a35835\"/\"distinct-w5-a35835-kserve-self-signed-certs\"" Oct 30 05:48:08.713736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.713649 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 05:48:08.763625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.763587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ztf6\" (UniqueName: \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.763625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.763632 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.763868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.763655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.763868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.763782 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.763976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.763897 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.764077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.764018 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865394 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865484 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865579 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ztf6\" (UniqueName: \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865623 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865649 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.865893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.865718 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.866081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.866005 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.866228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.866127 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.866228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.866210 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.868032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.868007 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.868267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.868231 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:08.893596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:08.893561 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ztf6\" (UniqueName: \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\") pod \"distinct-w5-a35835-kserve-69cbc4dffc-c6d9w\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:09.018344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.018305 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:09.356601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.356492 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 05:48:09.361136 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:48:09.361090 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3630cedd_62fe_4d9f_84f5_130d11293b39.slice/crio-674ee2e54c2130d5ab4c7831d9c8b0bedd33773d9721c968b84a38d4321e639c WatchSource:0}: Error finding container 674ee2e54c2130d5ab4c7831d9c8b0bedd33773d9721c968b84a38d4321e639c: Status 404 returned error can't find the container with id 674ee2e54c2130d5ab4c7831d9c8b0bedd33773d9721c968b84a38d4321e639c Oct 30 05:48:09.452442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.452399 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:09.452615 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.452475 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:09.844422 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.844377 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" event={"ID":"3630cedd-62fe-4d9f-84f5-130d11293b39","Type":"ContainerStarted","Data":"63f583613e79a264dd2b9e48001076a99f1f78ff87bf4ad259279725ba17f197"} Oct 30 05:48:09.844890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.844434 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" event={"ID":"3630cedd-62fe-4d9f-84f5-130d11293b39","Type":"ContainerStarted","Data":"674ee2e54c2130d5ab4c7831d9c8b0bedd33773d9721c968b84a38d4321e639c"} Oct 30 05:48:09.889366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:09.889314 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podStartSLOduration=1.889298439 podStartE2EDuration="1.889298439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:09.885028816 +0000 UTC m=+948.325406008" watchObservedRunningTime="2026-10-30 05:48:09.889298439 +0000 UTC m=+948.329675684" Oct 30 05:48:10.019404 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:10.019280 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:10.452813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:10.452773 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" start-of-body= Oct 30 05:48:10.453002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:10.452837 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": dial tcp 10.133.0.85:15021: connect: connection refused" Oct 30 05:48:11.840093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:11.840028 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:11.840531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:11.840364 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:11.841582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:11.841550 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 05:48:19.020071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:19.019399 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:19.047238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:19.047212 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:19.961235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:19.961205 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:48:20.750428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.750399 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:48:20.755109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.755082 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.785249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.785225 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:48:20.844585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844557 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844588 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844617 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844723 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844746 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844821 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844876 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.844991 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.844970 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.845027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.845014 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fdv\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946235 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946265 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946309 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946367 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946448 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946488 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fdv\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946532 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946554 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946597 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946687 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946875 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.946913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.946900 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.947373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.947292 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.949190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.949158 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.949373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.949349 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.975830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.975797 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fdv\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:20.977724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:20.977704 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:21.070454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.070393 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:21.395406 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:48:21.395378 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34715cfc_378c_4447_88a8_4c271e37c2f1.slice/crio-e956b9898a35d90067c33537de651f85d44c13722af912f6a5ebad2b2aeafff6 WatchSource:0}: Error finding container e956b9898a35d90067c33537de651f85d44c13722af912f6a5ebad2b2aeafff6: Status 404 returned error can't find the container with id e956b9898a35d90067c33537de651f85d44c13722af912f6a5ebad2b2aeafff6 Oct 30 05:48:21.396498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.396474 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:48:21.398166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.398131 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:21.398263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.398202 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:21.398263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.398231 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:21.967763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.967725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" event={"ID":"34715cfc-378c-4447-88a8-4c271e37c2f1","Type":"ContainerStarted","Data":"6bb8c43e6d7f849589c814a5dae99846f40e8c8316ceca295d16248573790acf"} Oct 30 05:48:21.968243 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:21.967786 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" event={"ID":"34715cfc-378c-4447-88a8-4c271e37c2f1","Type":"ContainerStarted","Data":"e956b9898a35d90067c33537de651f85d44c13722af912f6a5ebad2b2aeafff6"} Oct 30 05:48:22.020512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:22.020456 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podStartSLOduration=2.020443241 podStartE2EDuration="2.020443241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:22.018195053 +0000 UTC m=+960.458572247" watchObservedRunningTime="2026-10-30 05:48:22.020443241 +0000 UTC m=+960.460820433" Oct 30 05:48:22.071550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:22.071519 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:22.073227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:22.073205 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:22.073398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:22.073256 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:23.071374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:23.071342 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:23.071802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:23.071394 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:24.070812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:24.070782 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:24.071012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:24.070842 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:25.070800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:25.070764 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:25.071285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:25.070837 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:26.071490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:26.071459 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:26.071879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:26.071521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:26.388601 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:48:26.388495 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:48:27.071840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:27.071809 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:48:27.072249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:27.071862 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:48:27.553152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:27.553114 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:48:28.284282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:28.284248 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:28.284814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:28.284795 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:28.286368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:28.286347 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:48:36.126829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.126797 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 05:48:36.134612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.134578 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.159095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.158811 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 05:48:36.165593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.165562 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"distinct-2-w1-4a68a1-kserve-self-signed-certs\"" Oct 30 05:48:36.165758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.165622 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"default-dockercfg-576t9\"" Oct 30 05:48:36.171843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.171819 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"openshift-service-ca.crt\"" Oct 30 05:48:36.171843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.171819 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4a68a1\"/\"kube-root-ca.crt\"" Oct 30 05:48:36.175677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.175617 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.175677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.175657 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz7ff\" (UniqueName: \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.175937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.175727 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.176004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.175956 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.176004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.175983 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.176185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.176008 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.201019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.200991 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 05:48:36.205591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.205574 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.247697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.247660 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 05:48:36.277493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277451 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.277493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277509 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.277747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277646 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.277747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277686 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.277747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277720 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.277910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.277755 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278038 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxqg6\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278151 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278192 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278247 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278315 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278343 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz7ff\" (UniqueName: \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278412 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278440 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.278838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278750 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.278924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.278848 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.280485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.280466 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.280934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.280918 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.302931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.302899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz7ff\" (UniqueName: \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\") pod \"distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.379809 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.379877 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.379925 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.379960 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380000 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380158 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxqg6\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380192 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380221 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380327 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.380866 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.382113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.381250 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.393069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.391268 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.393069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.391551 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.393069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.391791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.406486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.404389 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.406663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.406540 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.420621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.420567 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxqg6\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.422498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.422469 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.455599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.455566 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:36.519937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.519847 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:36.778235 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:48:36.778198 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2ce5ad1_cd0d_41e5_a16c_b4f261b90788.slice/crio-fc4bb32bee99a4c96ca5df8084e0ffd564598c4689cf3ab5771f37fd352f27d9 WatchSource:0}: Error finding container fc4bb32bee99a4c96ca5df8084e0ffd564598c4689cf3ab5771f37fd352f27d9: Status 404 returned error can't find the container with id fc4bb32bee99a4c96ca5df8084e0ffd564598c4689cf3ab5771f37fd352f27d9 Oct 30 05:48:36.778731 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.778697 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 05:48:36.908537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.908502 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 05:48:36.919355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.919312 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:36.919492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.919472 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:36.919560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:36.919544 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:48:37.126343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.126309 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" event={"ID":"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788","Type":"ContainerStarted","Data":"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e"} Oct 30 05:48:37.126574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.126359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" event={"ID":"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788","Type":"ContainerStarted","Data":"fc4bb32bee99a4c96ca5df8084e0ffd564598c4689cf3ab5771f37fd352f27d9"} Oct 30 05:48:37.128124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.128095 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" event={"ID":"33f0043c-6834-4bf4-88a3-fdd1662b51ff","Type":"ContainerStarted","Data":"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331"} Oct 30 05:48:37.128457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.128143 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" event={"ID":"33f0043c-6834-4bf4-88a3-fdd1662b51ff","Type":"ContainerStarted","Data":"20c525c94c24614763da09164da3d66f0f04e92435222a03f72ce7c17a678b3a"} Oct 30 05:48:37.169495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.169402 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podStartSLOduration=1.169388921 podStartE2EDuration="1.169388921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:37.168708296 +0000 UTC m=+975.609085490" watchObservedRunningTime="2026-10-30 05:48:37.169388921 +0000 UTC m=+975.609766162" Oct 30 05:48:37.221748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.221688 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podStartSLOduration=1.221668976 podStartE2EDuration="1.221668976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:37.216979703 +0000 UTC m=+975.657356897" watchObservedRunningTime="2026-10-30 05:48:37.221668976 +0000 UTC m=+975.662046174" Oct 30 05:48:37.459127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.458152 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:37.520453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.520420 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:37.521752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.521724 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.521922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:37.521775 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:38.525074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:38.524486 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.525074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:38.524547 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:39.521605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:39.521560 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.521786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:39.521634 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:40.521570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:40.521528 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:40.521978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:40.521590 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:41.523538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:41.523499 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:41.524168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:41.524136 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:42.520786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:42.520749 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:42.520956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:42.520821 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:43.521547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:43.521512 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:43.521960 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:43.521583 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 05:48:45.506535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:45.506497 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:45.507108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:45.507084 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:45.614483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:45.614458 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 05:48:46.456269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:46.456241 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:46.469690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:46.469662 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:46.571737 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:46.571704 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:48:47.347743 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:47.347517 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:48:56.380847 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:48:56.380808 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:48:56.863083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.863053 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g"] Oct 30 05:48:56.868396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.868378 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:56.888221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.888198 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g"] Oct 30 05:48:56.895245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.895224 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:56.895522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.895490 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:56.895685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.895671 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:56.895838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.895822 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:48:56.936652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.936630 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4q45\" (UniqueName: \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:56.936807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.936787 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:56.936921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:56.936908 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.038223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.038198 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.038335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.038291 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.038335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.038314 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4q45\" (UniqueName: \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.038981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.038954 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.040699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.040679 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.061298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.061271 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4q45\" (UniqueName: \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\") pod \"maas-api-revoke-sub-71ab8229-n277g\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.183333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.183260 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:57.492892 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:48:57.492865 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1826fe_97dc_4c64_919e_031f3c73a714.slice/crio-5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff WatchSource:0}: Error finding container 5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff: Status 404 returned error can't find the container with id 5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff Oct 30 05:48:57.493244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:57.493073 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g"] Oct 30 05:48:58.385546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:58.385517 6171 generic.go:356] "Generic (PLEG): container finished" podID="af1826fe-97dc-4c64-919e-031f3c73a714" containerID="caf3f41c32faf1a9a2f3f41fb25a95e9d38a9752e29ddc658bde3507669d174b" exitCode=0 Oct 30 05:48:58.385662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:58.385594 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" event={"ID":"af1826fe-97dc-4c64-919e-031f3c73a714","Type":"ContainerDied","Data":"caf3f41c32faf1a9a2f3f41fb25a95e9d38a9752e29ddc658bde3507669d174b"} Oct 30 05:48:58.385662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:58.385648 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" event={"ID":"af1826fe-97dc-4c64-919e-031f3c73a714","Type":"ContainerStarted","Data":"5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff"} Oct 30 05:48:59.477576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.457446 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5"] Oct 30 05:48:59.477576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.465111 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.477576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.476883 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5"] Oct 30 05:48:59.488026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.487658 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8fvf\" (UniqueName: \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.488026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.487784 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.488026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.487867 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.589412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.589370 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.589961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.589560 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8fvf\" (UniqueName: \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.589961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.589657 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.590141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.589974 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.592332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.592308 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.612892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.612870 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8fvf\" (UniqueName: \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\") pod \"maas-api-revoke-sub-202061a5-q5mr5\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.638268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.638244 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:48:59.691480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.691215 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\" (UniqueName: \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\") pod \"af1826fe-97dc-4c64-919e-031f3c73a714\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " Oct 30 05:48:59.691480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.691356 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\") pod \"af1826fe-97dc-4c64-919e-031f3c73a714\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " Oct 30 05:48:59.691480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.691458 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\") pod \"af1826fe-97dc-4c64-919e-031f3c73a714\" (UID: \"af1826fe-97dc-4c64-919e-031f3c73a714\") " Oct 30 05:48:59.693474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.692251 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca" pod "af1826fe-97dc-4c64-919e-031f3c73a714" (UID: "af1826fe-97dc-4c64-919e-031f3c73a714"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:59.696316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.694505 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca" pod "af1826fe-97dc-4c64-919e-031f3c73a714" (UID: "af1826fe-97dc-4c64-919e-031f3c73a714"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:59.696316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.694765 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45" pod "af1826fe-97dc-4c64-919e-031f3c73a714" (UID: "af1826fe-97dc-4c64-919e-031f3c73a714"). InnerVolumeSpecName "kube-api-access-c4q45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:59.791468 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.791443 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:48:59.793109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.793091 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:48:59.793179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.793114 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1826fe-97dc-4c64-919e-031f3c73a714-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:48:59.793179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.793123 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4q45\" (UniqueName: \"kubernetes.io/projected/af1826fe-97dc-4c64-919e-031f3c73a714-kube-api-access-c4q45\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:48:59.939548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.939517 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8"] Oct 30 05:48:59.940213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.940197 6171 cpu_manager.go:414] "containerMap: removing container" podUID="af1826fe-97dc-4c64-919e-031f3c73a714" containerName="revoke-keys" Oct 30 05:48:59.940302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.940213 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1826fe-97dc-4c64-919e-031f3c73a714" containerName="revoke-keys" Oct 30 05:48:59.940361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.940306 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1826fe-97dc-4c64-919e-031f3c73a714" containerName="revoke-keys" Oct 30 05:48:59.945141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.945088 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:48:59.966493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.964681 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8"] Oct 30 05:48:59.995961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.995931 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:48:59.996138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.996010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l7dx\" (UniqueName: \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:48:59.996230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:48:59.996210 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.097147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.097115 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.097283 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.097216 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.097345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.097279 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l7dx\" (UniqueName: \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.097673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.097649 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5"] Oct 30 05:49:00.098039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.097987 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.098273 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:00.098253 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbc142e_6ead_491d_accf_9043427d13ac.slice/crio-f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a WatchSource:0}: Error finding container f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a: Status 404 returned error can't find the container with id f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a Oct 30 05:49:00.099779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.099748 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.121436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.121411 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l7dx\" (UniqueName: \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\") pod \"maas-api-revoke-sub-2bea5546-69mm8\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.260147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.260112 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:00.407135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.406469 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" event={"ID":"af1826fe-97dc-4c64-919e-031f3c73a714","Type":"ContainerDied","Data":"5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff"} Oct 30 05:49:00.407135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.406852 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b29396043e82c447968a950e5eb1c72bce175e9d48b4542ad9fe0bd65a4b2ff" Oct 30 05:49:00.407135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.406584 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g" Oct 30 05:49:00.413700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.413665 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" event={"ID":"cbbc142e-6ead-491d-accf-9043427d13ac","Type":"ContainerStarted","Data":"f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a"} Oct 30 05:49:00.458616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.458567 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" podStartSLOduration=1.458549632 podStartE2EDuration="1.458549632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:00.456318212 +0000 UTC m=+998.896695407" watchObservedRunningTime="2026-10-30 05:49:00.458549632 +0000 UTC m=+998.898926830" Oct 30 05:49:00.584398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:00.584375 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8"] Oct 30 05:49:00.584940 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:00.584921 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e8c1aa_66ee_4271_9057_45d9e5bb580f.slice/crio-edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3 WatchSource:0}: Error finding container edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3: Status 404 returned error can't find the container with id edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3 Oct 30 05:49:01.048068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.042724 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt"] Oct 30 05:49:01.061711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.061685 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt"] Oct 30 05:49:01.061863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.061779 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.132922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.132893 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.133094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.132928 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r784r\" (UniqueName: \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.133169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.133103 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.233896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.233867 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.234009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.233899 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r784r\" (UniqueName: \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.234009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.233982 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.234600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.234582 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.236347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.236325 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.257489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.257467 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r784r\" (UniqueName: \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\") pod \"maas-api-revoke-sub-60c04f47-phxjt\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.376093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.375998 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:01.424137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.424089 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" event={"ID":"a2e8c1aa-66ee-4271-9057-45d9e5bb580f","Type":"ContainerStarted","Data":"a16a349425efb2cbc6cc2e4ade0fabb8375e6d9bcd4bde80f12f2dfae113b7f3"} Oct 30 05:49:01.424333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.424160 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" event={"ID":"a2e8c1aa-66ee-4271-9057-45d9e5bb580f","Type":"ContainerStarted","Data":"edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3"} Oct 30 05:49:01.426251 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.426224 6171 generic.go:356] "Generic (PLEG): container finished" podID="cbbc142e-6ead-491d-accf-9043427d13ac" containerID="5fe68aa8699dd2901a0972fe44e8fcb20eac7d9bc33ada0f5a6c97ce5fd1c81e" exitCode=0 Oct 30 05:49:01.426357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.426284 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" event={"ID":"cbbc142e-6ead-491d-accf-9043427d13ac","Type":"ContainerDied","Data":"5fe68aa8699dd2901a0972fe44e8fcb20eac7d9bc33ada0f5a6c97ce5fd1c81e"} Oct 30 05:49:01.466787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.466729 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" podStartSLOduration=2.466716783 podStartE2EDuration="2.466716783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:01.463439287 +0000 UTC m=+999.903816484" watchObservedRunningTime="2026-10-30 05:49:01.466716783 +0000 UTC m=+999.907093977" Oct 30 05:49:01.733862 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:01.733836 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a17437_5bde_411e_abe8_08e255ec689b.slice/crio-327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5 WatchSource:0}: Error finding container 327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5: Status 404 returned error can't find the container with id 327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5 Oct 30 05:49:01.735293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:01.735269 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt"] Oct 30 05:49:02.437007 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.436918 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" event={"ID":"c3a17437-5bde-411e-abe8-08e255ec689b","Type":"ContainerStarted","Data":"f6f85c102e8b57b0ea8c81c29cd031425cf319e74230f6709c25d08fca3c9557"} Oct 30 05:49:02.437007 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.436983 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" event={"ID":"c3a17437-5bde-411e-abe8-08e255ec689b","Type":"ContainerStarted","Data":"327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5"} Oct 30 05:49:02.438633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.438598 6171 generic.go:356] "Generic (PLEG): container finished" podID="a2e8c1aa-66ee-4271-9057-45d9e5bb580f" containerID="a16a349425efb2cbc6cc2e4ade0fabb8375e6d9bcd4bde80f12f2dfae113b7f3" exitCode=0 Oct 30 05:49:02.438760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.438686 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" event={"ID":"a2e8c1aa-66ee-4271-9057-45d9e5bb580f","Type":"ContainerDied","Data":"a16a349425efb2cbc6cc2e4ade0fabb8375e6d9bcd4bde80f12f2dfae113b7f3"} Oct 30 05:49:02.485896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.485835 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" podStartSLOduration=2.485814434 podStartE2EDuration="2.485814434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:02.479538932 +0000 UTC m=+1000.919916125" watchObservedRunningTime="2026-10-30 05:49:02.485814434 +0000 UTC m=+1000.926191652" Oct 30 05:49:02.683276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.683247 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:49:02.757184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.757148 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\") pod \"cbbc142e-6ead-491d-accf-9043427d13ac\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " Oct 30 05:49:02.757664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.757256 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\") pod \"cbbc142e-6ead-491d-accf-9043427d13ac\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " Oct 30 05:49:02.757664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.757439 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\" (UniqueName: \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\") pod \"cbbc142e-6ead-491d-accf-9043427d13ac\" (UID: \"cbbc142e-6ead-491d-accf-9043427d13ac\") " Oct 30 05:49:02.758217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.758190 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca" pod "cbbc142e-6ead-491d-accf-9043427d13ac" (UID: "cbbc142e-6ead-491d-accf-9043427d13ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:02.758444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.758422 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:02.772129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.772090 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf" pod "cbbc142e-6ead-491d-accf-9043427d13ac" (UID: "cbbc142e-6ead-491d-accf-9043427d13ac"). InnerVolumeSpecName "kube-api-access-k8fvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:02.773297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.773272 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca" pod "cbbc142e-6ead-491d-accf-9043427d13ac" (UID: "cbbc142e-6ead-491d-accf-9043427d13ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:02.860444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.860365 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8fvf\" (UniqueName: \"kubernetes.io/projected/cbbc142e-6ead-491d-accf-9043427d13ac-kube-api-access-k8fvf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:02.860444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:02.860403 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbbc142e-6ead-491d-accf-9043427d13ac-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:03.448526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.448493 6171 generic.go:356] "Generic (PLEG): container finished" podID="c3a17437-5bde-411e-abe8-08e255ec689b" containerID="f6f85c102e8b57b0ea8c81c29cd031425cf319e74230f6709c25d08fca3c9557" exitCode=0 Oct 30 05:49:03.448701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.448570 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" event={"ID":"c3a17437-5bde-411e-abe8-08e255ec689b","Type":"ContainerDied","Data":"f6f85c102e8b57b0ea8c81c29cd031425cf319e74230f6709c25d08fca3c9557"} Oct 30 05:49:03.450236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.450205 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" event={"ID":"cbbc142e-6ead-491d-accf-9043427d13ac","Type":"ContainerDied","Data":"f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a"} Oct 30 05:49:03.450236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.450226 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5" Oct 30 05:49:03.450407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.450257 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3e78593d73f1222c127b1c37f03536c85a3fa12b6c4a57c3f63d81df9434d7a" Oct 30 05:49:03.675813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.675782 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:03.777473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.777444 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\") pod \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " Oct 30 05:49:03.777931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.777517 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\") pod \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " Oct 30 05:49:03.777931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.777575 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\" (UniqueName: \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\") pod \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\" (UID: \"a2e8c1aa-66ee-4271-9057-45d9e5bb580f\") " Oct 30 05:49:03.777931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.777887 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca" pod "a2e8c1aa-66ee-4271-9057-45d9e5bb580f" (UID: "a2e8c1aa-66ee-4271-9057-45d9e5bb580f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:03.778560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.778539 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:03.780823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.780798 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca" pod "a2e8c1aa-66ee-4271-9057-45d9e5bb580f" (UID: "a2e8c1aa-66ee-4271-9057-45d9e5bb580f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:03.780921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.780836 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx" pod "a2e8c1aa-66ee-4271-9057-45d9e5bb580f" (UID: "a2e8c1aa-66ee-4271-9057-45d9e5bb580f"). InnerVolumeSpecName "kube-api-access-6l7dx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:03.888070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.883373 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6l7dx\" (UniqueName: \"kubernetes.io/projected/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-kube-api-access-6l7dx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:03.888070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:03.883411 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2e8c1aa-66ee-4271-9057-45d9e5bb580f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:04.473080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.473006 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" Oct 30 05:49:04.473080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.473006 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8" event={"ID":"a2e8c1aa-66ee-4271-9057-45d9e5bb580f","Type":"ContainerDied","Data":"edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3"} Oct 30 05:49:04.473332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.473124 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edc58604e4e55e1ac1d6916457ad2c2d467921002f1f26bb13abe7f36a20f8b3" Oct 30 05:49:04.713560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.713530 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:04.802680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.802650 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\") pod \"c3a17437-5bde-411e-abe8-08e255ec689b\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " Oct 30 05:49:04.803170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.802772 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\" (UniqueName: \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\") pod \"c3a17437-5bde-411e-abe8-08e255ec689b\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " Oct 30 05:49:04.803170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.802824 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\") pod \"c3a17437-5bde-411e-abe8-08e255ec689b\" (UID: \"c3a17437-5bde-411e-abe8-08e255ec689b\") " Oct 30 05:49:04.803667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.803645 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca" pod "c3a17437-5bde-411e-abe8-08e255ec689b" (UID: "c3a17437-5bde-411e-abe8-08e255ec689b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:04.805338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.805315 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r" pod "c3a17437-5bde-411e-abe8-08e255ec689b" (UID: "c3a17437-5bde-411e-abe8-08e255ec689b"). InnerVolumeSpecName "kube-api-access-r784r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:04.805440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.805401 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca" pod "c3a17437-5bde-411e-abe8-08e255ec689b" (UID: "c3a17437-5bde-411e-abe8-08e255ec689b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:04.904875 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.904819 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:04.905089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.904892 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r784r\" (UniqueName: \"kubernetes.io/projected/c3a17437-5bde-411e-abe8-08e255ec689b-kube-api-access-r784r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:04.905089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:04.904908 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c3a17437-5bde-411e-abe8-08e255ec689b-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:05.291110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.290791 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 05:49:05.291819 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.291793 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a2e8c1aa-66ee-4271-9057-45d9e5bb580f" containerName="revoke-keys" Oct 30 05:49:05.292009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.291993 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2e8c1aa-66ee-4271-9057-45d9e5bb580f" containerName="revoke-keys" Oct 30 05:49:05.292155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292140 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c3a17437-5bde-411e-abe8-08e255ec689b" containerName="revoke-keys" Oct 30 05:49:05.292248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292236 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a17437-5bde-411e-abe8-08e255ec689b" containerName="revoke-keys" Oct 30 05:49:05.292363 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292352 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cbbc142e-6ead-491d-accf-9043427d13ac" containerName="revoke-keys" Oct 30 05:49:05.292459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292446 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbbc142e-6ead-491d-accf-9043427d13ac" containerName="revoke-keys" Oct 30 05:49:05.292777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292761 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2e8c1aa-66ee-4271-9057-45d9e5bb580f" containerName="revoke-keys" Oct 30 05:49:05.292886 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292875 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a17437-5bde-411e-abe8-08e255ec689b" containerName="revoke-keys" Oct 30 05:49:05.292970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.292961 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbbc142e-6ead-491d-accf-9043427d13ac" containerName="revoke-keys" Oct 30 05:49:05.299394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.299372 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.321725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.321699 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 05:49:05.420339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.420301 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.420793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.420770 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.422578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.422548 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.422736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.422716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.422870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.422772 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.422937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.422907 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.422992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.422944 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98xkw\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.423153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.423078 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.423239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.423175 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.496797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.496748 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" event={"ID":"c3a17437-5bde-411e-abe8-08e255ec689b","Type":"ContainerDied","Data":"327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5"} Oct 30 05:49:05.497071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.497029 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="327b985d1c4ff1317d97b9815ce04192f7d48f28319211d88cec0ed78f6210c5" Oct 30 05:49:05.497305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.497290 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt" Oct 30 05:49:05.523934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.523907 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.523963 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.523990 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524063 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524131 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524153 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98xkw\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.524506 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524330 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525143 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.525125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.525163 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524937 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524992 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.525038 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524499 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.525484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.524608 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.526594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.526573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.527969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.527945 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.547728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.547675 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98xkw\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.547910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.547888 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:05.635292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:05.635259 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:06.011093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.010529 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 05:49:06.049714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.047420 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:06.049714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.047512 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:06.049714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.047558 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:06.510373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.510075 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" event={"ID":"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a","Type":"ContainerStarted","Data":"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0"} Oct 30 05:49:06.510373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.510129 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" event={"ID":"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a","Type":"ContainerStarted","Data":"98fbc1dc44935e4587126637faf51a7d6ba5c81c86bb9a196a7815b2eddecd3e"} Oct 30 05:49:06.563743 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.563680 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podStartSLOduration=1.563660537 podStartE2EDuration="1.563660537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:06.560886824 +0000 UTC m=+1005.001264019" watchObservedRunningTime="2026-10-30 05:49:06.563660537 +0000 UTC m=+1005.004037730" Oct 30 05:49:06.640171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:06.640136 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:07.209793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:07.209758 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:49:07.637832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:07.637798 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:07.638036 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:07.637853 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:08.658813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:08.658171 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:08.658813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:08.658264 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:09.635564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:09.635531 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:09.635847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:09.635820 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:10.645562 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.645525 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:10.646321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.646267 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:10.678965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.678765 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:10.787455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.787414 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:10.846655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.846623 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd"] Oct 30 05:49:10.857473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.857447 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:10.945121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.898347 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:10.945121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.898435 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:10.945121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.899106 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:49:10.945121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:10.899144 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:11.055066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.002018 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.055066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.002262 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbhkz\" (UniqueName: \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.055066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.002293 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.073141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.073112 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd"] Oct 30 05:49:11.112584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.112545 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbhkz\" (UniqueName: \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.112841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.112817 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.113065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.113029 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.114118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.114095 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.124215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.124158 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.170909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.170869 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbhkz\" (UniqueName: \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\") pod \"maas-api-revoke-sub-c1ce8661-896xd\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.338434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.338399 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:11.554714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.554676 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p"] Oct 30 05:49:11.561562 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.561537 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.618259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.618181 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p"] Oct 30 05:49:11.636300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.636278 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:11.636600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.636528 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:11.643118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.642951 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.643118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.642994 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.643118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.643068 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h7tt\" (UniqueName: \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.735515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.735488 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd"] Oct 30 05:49:11.747197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.746667 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.747197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.746773 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h7tt\" (UniqueName: \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.747197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.746964 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.749010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.748988 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.753224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.753201 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.771270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.771247 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h7tt\" (UniqueName: \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\") pod \"maas-api-revoke-sub-88c73b74-f8d7p\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:11.904117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:11.899007 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:12.310515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.310482 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p"] Oct 30 05:49:12.636393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.636355 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:12.636564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.636403 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:12.668490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.668442 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" event={"ID":"8cc78904-1c9f-4eea-8195-d4980a8eef96","Type":"ContainerStarted","Data":"5f45cae5fa86384d1ddae35722713080f2946d003c3ef10b00abe0cf8a341b03"} Oct 30 05:49:12.668726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.668519 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" event={"ID":"8cc78904-1c9f-4eea-8195-d4980a8eef96","Type":"ContainerStarted","Data":"e91288f31e13e7052019d00e429e807f095e2d8029c790041d33fa3d2a66055e"} Oct 30 05:49:12.670647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.670622 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" event={"ID":"017cc91c-07b9-48ae-82ae-4e058165e19b","Type":"ContainerStarted","Data":"bfe5db1070f086e84297f95425324b1763c3d68fb0822bc0c14bb0e4c7dbf14f"} Oct 30 05:49:12.729217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:12.729176 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" podStartSLOduration=2.72915899 podStartE2EDuration="2.72915899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:12.726192316 +0000 UTC m=+1011.166569513" watchObservedRunningTime="2026-10-30 05:49:12.72915899 +0000 UTC m=+1011.169536183" Oct 30 05:49:13.395502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.395473 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d"] Oct 30 05:49:13.399876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.399854 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.420266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.420242 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d"] Oct 30 05:49:13.584329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.584299 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75db2\" (UniqueName: \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.584487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.584373 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.584533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.584497 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.635687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.635660 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" start-of-body= Oct 30 05:49:13.635800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.635711 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.94:15021/healthz/ready\": dial tcp 10.133.0.94:15021: connect: connection refused" Oct 30 05:49:13.691690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.691606 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75db2\" (UniqueName: \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.691889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.691763 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.691975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.691911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.692531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.692504 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.692950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.692926 6171 generic.go:356] "Generic (PLEG): container finished" podID="8cc78904-1c9f-4eea-8195-d4980a8eef96" containerID="5f45cae5fa86384d1ddae35722713080f2946d003c3ef10b00abe0cf8a341b03" exitCode=0 Oct 30 05:49:13.693117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.693096 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" event={"ID":"8cc78904-1c9f-4eea-8195-d4980a8eef96","Type":"ContainerDied","Data":"5f45cae5fa86384d1ddae35722713080f2946d003c3ef10b00abe0cf8a341b03"} Oct 30 05:49:13.695092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.695068 6171 generic.go:356] "Generic (PLEG): container finished" podID="017cc91c-07b9-48ae-82ae-4e058165e19b" containerID="9f4eab8b4245e508643baed56d9f98b3ef6272c94135837ab65830d5f253def6" exitCode=0 Oct 30 05:49:13.695188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.695168 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" event={"ID":"017cc91c-07b9-48ae-82ae-4e058165e19b","Type":"ContainerDied","Data":"9f4eab8b4245e508643baed56d9f98b3ef6272c94135837ab65830d5f253def6"} Oct 30 05:49:13.701412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.701389 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.746178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.746151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75db2\" (UniqueName: \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\") pod \"maas-api-revoke-sub-43f71aef-n4w8d\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:13.765997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:13.765976 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:14.083923 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:14.083882 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe0a44e6_ee45_4623_be40_4b90a9eab753.slice/crio-bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40 WatchSource:0}: Error finding container bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40: Status 404 returned error can't find the container with id bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40 Oct 30 05:49:14.087003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.086975 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d"] Oct 30 05:49:14.707107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.707009 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" event={"ID":"fe0a44e6-ee45-4623-be40-4b90a9eab753","Type":"ContainerStarted","Data":"8865d06e575fa5d4a529dd5da78ce95bbf94321dcc91875a4969d6e4e50d41e1"} Oct 30 05:49:14.707107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.707084 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" event={"ID":"fe0a44e6-ee45-4623-be40-4b90a9eab753","Type":"ContainerStarted","Data":"bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40"} Oct 30 05:49:14.925704 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.925678 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:14.966265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.965238 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:14.979345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:14.979280 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" podStartSLOduration=1.979263237 podStartE2EDuration="1.979263237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:14.748346809 +0000 UTC m=+1013.188724005" watchObservedRunningTime="2026-10-30 05:49:14.979263237 +0000 UTC m=+1013.419640438" Oct 30 05:49:15.020456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.020417 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\" (UniqueName: \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\") pod \"017cc91c-07b9-48ae-82ae-4e058165e19b\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.020995 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\" (UniqueName: \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\") pod \"8cc78904-1c9f-4eea-8195-d4980a8eef96\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.021250 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\") pod \"8cc78904-1c9f-4eea-8195-d4980a8eef96\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.021403 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\") pod \"8cc78904-1c9f-4eea-8195-d4980a8eef96\" (UID: \"8cc78904-1c9f-4eea-8195-d4980a8eef96\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.021447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\") pod \"017cc91c-07b9-48ae-82ae-4e058165e19b\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.021610 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\") pod \"017cc91c-07b9-48ae-82ae-4e058165e19b\" (UID: \"017cc91c-07b9-48ae-82ae-4e058165e19b\") " Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.022245 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca" pod "8cc78904-1c9f-4eea-8195-d4980a8eef96" (UID: "8cc78904-1c9f-4eea-8195-d4980a8eef96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.022669 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca" pod "017cc91c-07b9-48ae-82ae-4e058165e19b" (UID: "017cc91c-07b9-48ae-82ae-4e058165e19b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.023557 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.029019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.023580 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.031371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.031335 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt" pod "017cc91c-07b9-48ae-82ae-4e058165e19b" (UID: "017cc91c-07b9-48ae-82ae-4e058165e19b"). InnerVolumeSpecName "kube-api-access-6h7tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:15.032907 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.032873 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca" pod "8cc78904-1c9f-4eea-8195-d4980a8eef96" (UID: "8cc78904-1c9f-4eea-8195-d4980a8eef96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:15.033097 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.033071 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz" pod "8cc78904-1c9f-4eea-8195-d4980a8eef96" (UID: "8cc78904-1c9f-4eea-8195-d4980a8eef96"). InnerVolumeSpecName "kube-api-access-kbhkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:15.036142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.036109 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca" pod "017cc91c-07b9-48ae-82ae-4e058165e19b" (UID: "017cc91c-07b9-48ae-82ae-4e058165e19b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:15.125653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.125614 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h7tt\" (UniqueName: \"kubernetes.io/projected/017cc91c-07b9-48ae-82ae-4e058165e19b-kube-api-access-6h7tt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.125653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.125649 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbhkz\" (UniqueName: \"kubernetes.io/projected/8cc78904-1c9f-4eea-8195-d4980a8eef96-kube-api-access-kbhkz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.125887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.125665 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cc78904-1c9f-4eea-8195-d4980a8eef96-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.125887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.125678 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/017cc91c-07b9-48ae-82ae-4e058165e19b-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:15.200015 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.199982 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:15.200535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.200511 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:15.201764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.201741 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 05:49:15.716756 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.716727 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" Oct 30 05:49:15.717241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.716727 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p" event={"ID":"017cc91c-07b9-48ae-82ae-4e058165e19b","Type":"ContainerDied","Data":"bfe5db1070f086e84297f95425324b1763c3d68fb0822bc0c14bb0e4c7dbf14f"} Oct 30 05:49:15.717241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.716852 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfe5db1070f086e84297f95425324b1763c3d68fb0822bc0c14bb0e4c7dbf14f" Oct 30 05:49:15.718878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.718804 6171 generic.go:356] "Generic (PLEG): container finished" podID="fe0a44e6-ee45-4623-be40-4b90a9eab753" containerID="8865d06e575fa5d4a529dd5da78ce95bbf94321dcc91875a4969d6e4e50d41e1" exitCode=0 Oct 30 05:49:15.719424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.719344 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" event={"ID":"fe0a44e6-ee45-4623-be40-4b90a9eab753","Type":"ContainerDied","Data":"8865d06e575fa5d4a529dd5da78ce95bbf94321dcc91875a4969d6e4e50d41e1"} Oct 30 05:49:15.721152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.721134 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" Oct 30 05:49:15.721261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.721170 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd" event={"ID":"8cc78904-1c9f-4eea-8195-d4980a8eef96","Type":"ContainerDied","Data":"e91288f31e13e7052019d00e429e807f095e2d8029c790041d33fa3d2a66055e"} Oct 30 05:49:15.721261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:15.721209 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e91288f31e13e7052019d00e429e807f095e2d8029c790041d33fa3d2a66055e" Oct 30 05:49:16.881112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.881082 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:16.956301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.956271 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\") pod \"fe0a44e6-ee45-4623-be40-4b90a9eab753\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " Oct 30 05:49:16.956521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.956323 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\") pod \"fe0a44e6-ee45-4623-be40-4b90a9eab753\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " Oct 30 05:49:16.956521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.956453 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\" (UniqueName: \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\") pod \"fe0a44e6-ee45-4623-be40-4b90a9eab753\" (UID: \"fe0a44e6-ee45-4623-be40-4b90a9eab753\") " Oct 30 05:49:16.956717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.956697 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca" pod "fe0a44e6-ee45-4623-be40-4b90a9eab753" (UID: "fe0a44e6-ee45-4623-be40-4b90a9eab753"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:16.957058 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.957009 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:16.958909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.958874 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2" pod "fe0a44e6-ee45-4623-be40-4b90a9eab753" (UID: "fe0a44e6-ee45-4623-be40-4b90a9eab753"). InnerVolumeSpecName "kube-api-access-75db2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:16.959026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:16.958973 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca" pod "fe0a44e6-ee45-4623-be40-4b90a9eab753" (UID: "fe0a44e6-ee45-4623-be40-4b90a9eab753"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:17.058095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:17.058000 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe0a44e6-ee45-4623-be40-4b90a9eab753-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:17.058095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:17.058030 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75db2\" (UniqueName: \"kubernetes.io/projected/fe0a44e6-ee45-4623-be40-4b90a9eab753-kube-api-access-75db2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:17.738070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:17.738020 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" Oct 30 05:49:17.738242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:17.738019 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d" event={"ID":"fe0a44e6-ee45-4623-be40-4b90a9eab753","Type":"ContainerDied","Data":"bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40"} Oct 30 05:49:17.738242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:17.738156 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcb61d5bc15a2731d2ecb19266a82fd1e19a961c55d0fbac8b5d3ca59e91ae40" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.041177 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz"] Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042081 6171 cpu_manager.go:414] "containerMap: removing container" podUID="017cc91c-07b9-48ae-82ae-4e058165e19b" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042097 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="017cc91c-07b9-48ae-82ae-4e058165e19b" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042114 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fe0a44e6-ee45-4623-be40-4b90a9eab753" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042122 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe0a44e6-ee45-4623-be40-4b90a9eab753" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042154 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8cc78904-1c9f-4eea-8195-d4980a8eef96" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042161 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cc78904-1c9f-4eea-8195-d4980a8eef96" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042360 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="017cc91c-07b9-48ae-82ae-4e058165e19b" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042372 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe0a44e6-ee45-4623-be40-4b90a9eab753" containerName="revoke-keys" Oct 30 05:49:26.052104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.042384 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cc78904-1c9f-4eea-8195-d4980a8eef96" containerName="revoke-keys" Oct 30 05:49:26.064364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.062815 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz"] Oct 30 05:49:26.064364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.062929 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.142677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.078883 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:26.218084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.169509 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:26.218084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.169579 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:26.218084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.169664 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:49:26.280087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.267942 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cv89\" (UniqueName: \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.280087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.268024 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.280087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.268186 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.415126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.411528 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.415126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.411783 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cv89\" (UniqueName: \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.415126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.411843 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.415126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.412787 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.474069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.473712 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cv89\" (UniqueName: \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.495069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.487772 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\") pod \"maas-api-revoke-sub-168ced75-rrqtz\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.618072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:26.617481 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:26.650066 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:49:26.644963 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:49:27.728072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:27.727501 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:49:28.098067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:28.089649 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz"] Oct 30 05:49:28.358113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:28.353158 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:29.208741 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:29.208708 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" event={"ID":"6562722b-3a60-4fd5-89cf-c6a18dd92494","Type":"ContainerStarted","Data":"bc8ce0a370546ead4dbd37b8abc0c0cf27e52fc4b5eb71ad64bf332b1482cab1"} Oct 30 05:49:29.209150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:29.208759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" event={"ID":"6562722b-3a60-4fd5-89cf-c6a18dd92494","Type":"ContainerStarted","Data":"1fd3035c975b0c75481178fe0a09c40e9f2b384839f993b045bfdeb43248a2ea"} Oct 30 05:49:29.250789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:29.250737 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" podStartSLOduration=4.250724927 podStartE2EDuration="4.250724927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:29.247588253 +0000 UTC m=+1027.687965446" watchObservedRunningTime="2026-10-30 05:49:29.250724927 +0000 UTC m=+1027.691102120" Oct 30 05:49:30.221823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:30.221790 6171 generic.go:356] "Generic (PLEG): container finished" podID="6562722b-3a60-4fd5-89cf-c6a18dd92494" containerID="bc8ce0a370546ead4dbd37b8abc0c0cf27e52fc4b5eb71ad64bf332b1482cab1" exitCode=0 Oct 30 05:49:30.222311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:30.221883 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" event={"ID":"6562722b-3a60-4fd5-89cf-c6a18dd92494","Type":"ContainerDied","Data":"bc8ce0a370546ead4dbd37b8abc0c0cf27e52fc4b5eb71ad64bf332b1482cab1"} Oct 30 05:49:31.396721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.396697 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:31.542128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.541508 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\" (UniqueName: \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\") pod \"6562722b-3a60-4fd5-89cf-c6a18dd92494\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " Oct 30 05:49:31.542128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.541732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\") pod \"6562722b-3a60-4fd5-89cf-c6a18dd92494\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " Oct 30 05:49:31.542128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.541847 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\") pod \"6562722b-3a60-4fd5-89cf-c6a18dd92494\" (UID: \"6562722b-3a60-4fd5-89cf-c6a18dd92494\") " Oct 30 05:49:31.544519 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.543446 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca" pod "6562722b-3a60-4fd5-89cf-c6a18dd92494" (UID: "6562722b-3a60-4fd5-89cf-c6a18dd92494"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:31.564500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.564453 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca" pod "6562722b-3a60-4fd5-89cf-c6a18dd92494" (UID: "6562722b-3a60-4fd5-89cf-c6a18dd92494"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:31.566561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.566529 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89" pod "6562722b-3a60-4fd5-89cf-c6a18dd92494" (UID: "6562722b-3a60-4fd5-89cf-c6a18dd92494"). InnerVolumeSpecName "kube-api-access-5cv89". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:31.643523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.643492 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5cv89\" (UniqueName: \"kubernetes.io/projected/6562722b-3a60-4fd5-89cf-c6a18dd92494-kube-api-access-5cv89\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:31.643523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.643516 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:31.643523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:31.643528 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6562722b-3a60-4fd5-89cf-c6a18dd92494-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:32.239729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:32.239704 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" Oct 30 05:49:32.239729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:32.239708 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz" event={"ID":"6562722b-3a60-4fd5-89cf-c6a18dd92494","Type":"ContainerDied","Data":"1fd3035c975b0c75481178fe0a09c40e9f2b384839f993b045bfdeb43248a2ea"} Oct 30 05:49:32.239945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:32.239768 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fd3035c975b0c75481178fe0a09c40e9f2b384839f993b045bfdeb43248a2ea" Oct 30 05:49:34.326203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.326160 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js"] Oct 30 05:49:34.326930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.326913 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6562722b-3a60-4fd5-89cf-c6a18dd92494" containerName="revoke-keys" Oct 30 05:49:34.327017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.326932 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6562722b-3a60-4fd5-89cf-c6a18dd92494" containerName="revoke-keys" Oct 30 05:49:34.327084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.327056 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6562722b-3a60-4fd5-89cf-c6a18dd92494" containerName="revoke-keys" Oct 30 05:49:34.330780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.330762 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.345358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.345327 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js"] Oct 30 05:49:34.347353 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.347324 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:34.347353 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.347332 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:49:34.357223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.357200 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:34.357223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.357218 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:34.483767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.483738 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.483920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.483797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgrxh\" (UniqueName: \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.484074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.484026 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.585435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.585367 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.585435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.585408 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgrxh\" (UniqueName: \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.585624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.585520 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.586135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.586116 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.587872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.587851 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.608620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.608598 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgrxh\" (UniqueName: \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\") pod \"maas-api-revoke-sub-eef45f53-984js\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.647559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.647537 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:34.960526 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:34.960494 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21cad0fa_599f_4bdd_a41c_c4c93c97b83d.slice/crio-6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794 WatchSource:0}: Error finding container 6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794: Status 404 returned error can't find the container with id 6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794 Oct 30 05:49:34.960688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:34.960635 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js"] Oct 30 05:49:35.262655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.262631 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" event={"ID":"21cad0fa-599f-4bdd-a41c-c4c93c97b83d","Type":"ContainerStarted","Data":"6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794"} Oct 30 05:49:35.372930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.372898 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:49:35.377399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.377382 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.466847 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.466915 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.466996 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467156 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467234 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k7f\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467289 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467332 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467386 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.467779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.467411 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.471570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.471546 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:49:35.568786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.568710 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.568938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.568793 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.568938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.568834 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k7f\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.568938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.568866 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.568983 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569265 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569269 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569341 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569432 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569343 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569464 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569714 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569730 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.569994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.569976 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.571423 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.571401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.571533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.571431 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.592535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.592511 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.592633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.592588 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k7f\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\") pod \"e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:35.774751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:35.774722 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:36.059361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.059331 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:49:36.063562 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.063544 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.073894 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.073870 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.073903 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.073925 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074069 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074206 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074378 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074472 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074434 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.074642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.074618 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.102884 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.102811 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:49:36.117833 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.117594 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:49:36.119970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.119929 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.120112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.120019 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.120186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.120123 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.175389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175369 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175417 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175449 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175479 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175519 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175676 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175707 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175743 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175768 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175785 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.175858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175803 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.176123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.175870 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.176455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.176432 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.176592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.176573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.177778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.177757 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.178256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.178237 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.198982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.198957 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.199168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.199149 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.274137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.274098 6171 generic.go:356] "Generic (PLEG): container finished" podID="21cad0fa-599f-4bdd-a41c-c4c93c97b83d" containerID="d8e04a79e4dd899655a2c5443074dab745fd5227a776c952596481bc130e7f89" exitCode=0 Oct 30 05:49:36.274329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.274184 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" event={"ID":"21cad0fa-599f-4bdd-a41c-c4c93c97b83d","Type":"ContainerDied","Data":"d8e04a79e4dd899655a2c5443074dab745fd5227a776c952596481bc130e7f89"} Oct 30 05:49:36.276204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.276178 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" event={"ID":"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8","Type":"ContainerStarted","Data":"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644"} Oct 30 05:49:36.276342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.276219 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" event={"ID":"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8","Type":"ContainerStarted","Data":"fa0fcdf4152b7a5f7275527c1c7c22e2cc99496314a62311fedd7abebb8fa187"} Oct 30 05:49:36.366301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.366206 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podStartSLOduration=1.366193784 podStartE2EDuration="1.366193784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:36.363166825 +0000 UTC m=+1034.803544020" watchObservedRunningTime="2026-10-30 05:49:36.366193784 +0000 UTC m=+1034.806570976" Oct 30 05:49:36.379142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.378609 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:36.382791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.382765 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:49:36.401344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.401316 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.412154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.406733 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:49:36.412427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.412392 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.415198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.415172 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:49:36.431985 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.431952 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:49:36.490082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.489982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490236 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp2wl\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490292 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4hpw\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490321 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490378 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490526 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490553 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490576 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490608 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490629 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490656 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490688 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490742 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490766 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490843 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490865 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.491073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.490916 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.591967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.591934 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.591967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.591972 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.591989 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592021 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592063 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592088 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592120 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592147 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592171 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592193 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592279 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592305 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592354 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.592650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592722 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.592836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.592776 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.593210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593155 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593163 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593167 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593220 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.593475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593319 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp2wl\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593364 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4hpw\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.593645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593482 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593397 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.593757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.593667 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.594200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.594005 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.594364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.594342 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.594707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.594681 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.596235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.596196 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.596768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.596748 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.597527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.597480 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.598008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.597976 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.616775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.616670 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.617275 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.617256 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.617814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.617795 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4hpw\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.618496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.618480 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp2wl\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\") pod \"e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.738096 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:36.738013 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff63842_10f8_4757_9b7f_7b8d89c49e5b.slice/crio-5bba0503e1c1cc02b3ae8095402b546185347ff4c47ddfdf7e281b114fea802f WatchSource:0}: Error finding container 5bba0503e1c1cc02b3ae8095402b546185347ff4c47ddfdf7e281b114fea802f: Status 404 returned error can't find the container with id 5bba0503e1c1cc02b3ae8095402b546185347ff4c47ddfdf7e281b114fea802f Oct 30 05:49:36.739625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.739592 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:49:36.740993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.740954 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.741112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.741093 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.741155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.741143 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:36.768158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.768137 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:36.771858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.771836 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:36.775401 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:36.775378 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:37.116738 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:37.116700 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99c9d95e_7e51_4569_bd0b_e01bf5378d53.slice/crio-b266dd782f8d930a5cee1489b6b124c6093c814b327ec92ac526b24a299c11ed WatchSource:0}: Error finding container b266dd782f8d930a5cee1489b6b124c6093c814b327ec92ac526b24a299c11ed: Status 404 returned error can't find the container with id b266dd782f8d930a5cee1489b6b124c6093c814b327ec92ac526b24a299c11ed Oct 30 05:49:37.118092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.118063 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:49:37.119305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.119269 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.119367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.119350 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.119412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.119401 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.129163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.129142 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:49:37.134074 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:37.134036 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2a59f97_740c_42a0_a9fa_29e7eb665bd7.slice/crio-2739eebf36aff604ac2846a8c2a90dfc27553df5f64ab4fd4641d91c0e7e2bcd WatchSource:0}: Error finding container 2739eebf36aff604ac2846a8c2a90dfc27553df5f64ab4fd4641d91c0e7e2bcd: Status 404 returned error can't find the container with id 2739eebf36aff604ac2846a8c2a90dfc27553df5f64ab4fd4641d91c0e7e2bcd Oct 30 05:49:37.136911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.136884 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.137005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.136951 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.137005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.136980 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:37.360175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.360023 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" event={"ID":"99c9d95e-7e51-4569-bd0b-e01bf5378d53","Type":"ContainerStarted","Data":"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8"} Oct 30 05:49:37.360175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.360094 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" event={"ID":"99c9d95e-7e51-4569-bd0b-e01bf5378d53","Type":"ContainerStarted","Data":"b266dd782f8d930a5cee1489b6b124c6093c814b327ec92ac526b24a299c11ed"} Oct 30 05:49:37.364071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.363925 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" event={"ID":"c2a59f97-740c-42a0-a9fa-29e7eb665bd7","Type":"ContainerStarted","Data":"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b"} Oct 30 05:49:37.364071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.363975 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" event={"ID":"c2a59f97-740c-42a0-a9fa-29e7eb665bd7","Type":"ContainerStarted","Data":"2739eebf36aff604ac2846a8c2a90dfc27553df5f64ab4fd4641d91c0e7e2bcd"} Oct 30 05:49:37.369591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.369543 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" event={"ID":"aff63842-10f8-4757-9b7f-7b8d89c49e5b","Type":"ContainerStarted","Data":"8d6d9683bb042592d53ec32edc4064b8d9587ec68f53e0460338d1dcc3b11e82"} Oct 30 05:49:37.369720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.369592 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" event={"ID":"aff63842-10f8-4757-9b7f-7b8d89c49e5b","Type":"ContainerStarted","Data":"5bba0503e1c1cc02b3ae8095402b546185347ff4c47ddfdf7e281b114fea802f"} Oct 30 05:49:37.380801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.379530 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:37.385229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.384813 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:37.385229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.384865 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:37.460069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.459427 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podStartSLOduration=1.459408268 podStartE2EDuration="1.459408268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:37.457622573 +0000 UTC m=+1035.897999767" watchObservedRunningTime="2026-10-30 05:49:37.459408268 +0000 UTC m=+1035.899785463" Oct 30 05:49:37.463073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.461478 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podStartSLOduration=1.461460922 podStartE2EDuration="1.461460922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:37.411353201 +0000 UTC m=+1035.851730407" watchObservedRunningTime="2026-10-30 05:49:37.461460922 +0000 UTC m=+1035.901838117" Oct 30 05:49:37.768282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.768236 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:37.774789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.774760 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:37.776357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.776330 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:37.776585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.776561 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:37.955357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:37.955322 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:38.015697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.007852 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podStartSLOduration=2.007830579 podStartE2EDuration="2.007830579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:37.516897002 +0000 UTC m=+1035.957274218" watchObservedRunningTime="2026-10-30 05:49:38.007830579 +0000 UTC m=+1036.448207778" Oct 30 05:49:38.065118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.017603 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\") pod \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " Oct 30 05:49:38.065118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.062409 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\") pod \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " Oct 30 05:49:38.065118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.062473 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\" (UniqueName: \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\") pod \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\" (UID: \"21cad0fa-599f-4bdd-a41c-c4c93c97b83d\") " Oct 30 05:49:38.065415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.028687 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca" pod "21cad0fa-599f-4bdd-a41c-c4c93c97b83d" (UID: "21cad0fa-599f-4bdd-a41c-c4c93c97b83d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:38.070917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.070858 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh" pod "21cad0fa-599f-4bdd-a41c-c4c93c97b83d" (UID: "21cad0fa-599f-4bdd-a41c-c4c93c97b83d"). InnerVolumeSpecName "kube-api-access-sgrxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:38.074334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.074303 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca" pod "21cad0fa-599f-4bdd-a41c-c4c93c97b83d" (UID: "21cad0fa-599f-4bdd-a41c-c4c93c97b83d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:38.165966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.165148 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:38.166175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.166007 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:38.166175 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.166030 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgrxh\" (UniqueName: \"kubernetes.io/projected/21cad0fa-599f-4bdd-a41c-c4c93c97b83d-kube-api-access-sgrxh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:38.395136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.394457 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.395136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.394517 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:38.492085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.485955 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" Oct 30 05:49:38.521076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.495485 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js" event={"ID":"21cad0fa-599f-4bdd-a41c-c4c93c97b83d","Type":"ContainerDied","Data":"6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794"} Oct 30 05:49:38.812070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.792034 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f757d44aa8fac0f943222ba692882c5868a1da3a25678579975df66d86a5794" Oct 30 05:49:38.815911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.815873 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.816186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.816156 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:38.883082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.882406 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:38.883082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:38.882480 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:39.032028 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.031974 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.032213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.032072 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:39.382490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.382444 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.382789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.382752 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:39.774072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.771618 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.774072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.771678 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:39.774072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.772329 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.774072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.772370 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:39.794077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.776309 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:39.794077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.776349 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:39.828071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.827663 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.898072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.888296 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.898072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.889962 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.903185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.903134 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:39.958497 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:39.958458 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.031265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.031154 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.097243 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.097187 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:49:40.330124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.323302 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:49:40.380075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.379411 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.380075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.379473 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.785823 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.785888 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.786460 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.786509 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.786542 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:40.788070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.786596 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:40.852071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.843777 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.852071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.845657 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:49:40.954071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.911631 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.954709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.954683 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:49:40.955577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.949327 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.956304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.956286 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:49:40.997071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.981364 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:40.997670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:40.997641 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:49:41.045294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.045194 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:41.053070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.049251 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:49:41.345757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.345665 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:49:41.368079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.357812 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:49:41.407880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.406961 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:41.407880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.407023 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:41.919200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.919149 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.945368 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.932802 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.977221 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.977303 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.991491 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.017071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:41.991641 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:42.025180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.025131 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:49:42.079070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.071286 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.079070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.071344 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:42.079070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.072236 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:42.325069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.324425 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s"] Oct 30 05:49:42.326075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.325471 6171 cpu_manager.go:414] "containerMap: removing container" podUID="21cad0fa-599f-4bdd-a41c-c4c93c97b83d" containerName="revoke-keys" Oct 30 05:49:42.326075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.325495 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cad0fa-599f-4bdd-a41c-c4c93c97b83d" containerName="revoke-keys" Oct 30 05:49:42.326075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.325733 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cad0fa-599f-4bdd-a41c-c4c93c97b83d" containerName="revoke-keys" Oct 30 05:49:42.338070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.336704 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.358079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.357843 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:42.363068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.358830 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:49:42.363068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.358892 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:42.373074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.370344 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:42.373074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.370480 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:42.389067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.386391 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.389067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.386440 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:42.437070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.436790 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s"] Oct 30 05:49:42.441069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.438232 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.441069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.438281 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.441069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.438403 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgcbx\" (UniqueName: \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.546071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.545067 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.546071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.545230 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgcbx\" (UniqueName: \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.546071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.545466 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.546385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.546330 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.571071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.570504 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.616629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.600775 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgcbx\" (UniqueName: \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\") pod \"maas-api-revoke-sub-d3872f64-wjb5s\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:42.759415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.759350 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:42.781596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.781561 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.782016 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.781982 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:42.782445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.782132 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.782821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.782798 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:42.782972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.782564 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:42.783279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.783260 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:42.814841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:42.814809 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:43.039158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.039091 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:43.064624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.048139 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:49:43.066635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.048188 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:43.067185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.048240 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:43.095532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.080021 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:49:43.359229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.359171 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:43.384835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.384802 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.385112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.385082 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:43.514589 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.514563 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:49:43.771081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.769467 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.771081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.769522 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:43.920085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.880282 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.920085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.880337 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:43.955081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.943363 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:43.955081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:43.943421 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:44.181122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.107077 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:49:44.208662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.108336 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.209221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.108368 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:49:44.209655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.108399 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.473121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.471243 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:49:44.476295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.475958 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:44.476655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.476627 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:44.585600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.585435 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:49:44.727726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.727658 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s"] Oct 30 05:49:44.760350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.760321 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:49:44.971458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.971411 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:49:44.973274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.973155 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:44.973274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.973218 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:44.973543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.973283 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body= Oct 30 05:49:44.973543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.973324 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" Oct 30 05:49:44.976869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.976759 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:44.976869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:44.976818 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:45.137608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.137578 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:49:45.395071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.391475 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" start-of-body= Oct 30 05:49:45.411298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.411251 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": dial tcp 10.133.0.101:15021: connect: connection refused" Oct 30 05:49:45.589525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.589435 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" event={"ID":"77e4d840-8240-45e1-8daf-b70feb18e4b1","Type":"ContainerStarted","Data":"75cb1bb940741d3185018cda7addee9b39bc5913c0d15d57bb19582ddf390d76"} Oct 30 05:49:45.589762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.589740 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" event={"ID":"77e4d840-8240-45e1-8daf-b70feb18e4b1","Type":"ContainerStarted","Data":"ac8faff4cea761336b6c8ac124d014f4f6b71e73e74c4f2bc5e5c481d41e6110"} Oct 30 05:49:45.635188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.635134 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" podStartSLOduration=4.635116888 podStartE2EDuration="4.635116888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:45.631336416 +0000 UTC m=+1044.071713609" watchObservedRunningTime="2026-10-30 05:49:45.635116888 +0000 UTC m=+1044.075494083" Oct 30 05:49:45.769011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.768977 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" start-of-body= Oct 30 05:49:45.769224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.769038 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.102:15021/healthz/ready\": dial tcp 10.133.0.102:15021: connect: connection refused" Oct 30 05:49:45.772390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.772366 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" start-of-body= Oct 30 05:49:45.772454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.772423 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": dial tcp 10.133.0.103:15021: connect: connection refused" Oct 30 05:49:45.839914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.839829 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:45.840714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.840687 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:45.841791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:45.841757 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:49:46.663416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.663381 6171 generic.go:356] "Generic (PLEG): container finished" podID="77e4d840-8240-45e1-8daf-b70feb18e4b1" containerID="75cb1bb940741d3185018cda7addee9b39bc5913c0d15d57bb19582ddf390d76" exitCode=0 Oct 30 05:49:46.665416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.665387 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" event={"ID":"77e4d840-8240-45e1-8daf-b70feb18e4b1","Type":"ContainerDied","Data":"75cb1bb940741d3185018cda7addee9b39bc5913c0d15d57bb19582ddf390d76"} Oct 30 05:49:46.786116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.783302 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx"] Oct 30 05:49:46.791389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.791363 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:46.814462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.813307 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx"] Oct 30 05:49:46.839411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.839380 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:46.840715 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.840695 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:46.872162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.872131 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:49:46.921864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.921758 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:46.921864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.921809 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h9fs\" (UniqueName: \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:46.921864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:46.921848 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.023935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.023895 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.024201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.023950 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h9fs\" (UniqueName: \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.024201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.023993 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.025611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.025587 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.031313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.031258 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.066156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.066125 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h9fs\" (UniqueName: \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\") pod \"maas-api-revoke-sub-c69d5760-jkfcx\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.123227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.123195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:47.123999 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.123975 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:47.125034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.125014 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:49:47.134466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.134439 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:47.264398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.263769 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:47.265492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.265467 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:47.266659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.266567 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:49:47.451425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.451399 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx"] Oct 30 05:49:47.453779 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:47.453751 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff4a50f0_158c_4c58_bd8e_19eaa1a94ac5.slice/crio-e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b WatchSource:0}: Error finding container e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b: Status 404 returned error can't find the container with id e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b Oct 30 05:49:47.458376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.458354 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s"] Oct 30 05:49:47.464426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.464395 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.483220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.483196 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s"] Oct 30 05:49:47.542902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.542875 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.543117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.543003 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.543117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.543036 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddlf4\" (UniqueName: \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.551970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.551935 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:49:47.644081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.644011 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.644217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.644088 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddlf4\" (UniqueName: \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.644332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.644314 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.644834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.644812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.646786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.646762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.670934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.670909 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddlf4\" (UniqueName: \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\") pod \"maas-api-revoke-sub-05036bc6-gv94s\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.678807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.678786 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" event={"ID":"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5","Type":"ContainerStarted","Data":"e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b"} Oct 30 05:49:47.810413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.810380 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:47.814610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.814590 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:47.846604 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.846577 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\") pod \"77e4d840-8240-45e1-8daf-b70feb18e4b1\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " Oct 30 05:49:47.846730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.846644 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\") pod \"77e4d840-8240-45e1-8daf-b70feb18e4b1\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " Oct 30 05:49:47.846793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.846747 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\" (UniqueName: \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\") pod \"77e4d840-8240-45e1-8daf-b70feb18e4b1\" (UID: \"77e4d840-8240-45e1-8daf-b70feb18e4b1\") " Oct 30 05:49:47.846943 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.846920 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca" pod "77e4d840-8240-45e1-8daf-b70feb18e4b1" (UID: "77e4d840-8240-45e1-8daf-b70feb18e4b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:47.847320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.847296 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:47.848931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.848911 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca" pod "77e4d840-8240-45e1-8daf-b70feb18e4b1" (UID: "77e4d840-8240-45e1-8daf-b70feb18e4b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:47.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.849063 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx" pod "77e4d840-8240-45e1-8daf-b70feb18e4b1" (UID: "77e4d840-8240-45e1-8daf-b70feb18e4b1"). InnerVolumeSpecName "kube-api-access-dgcbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:47.948206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.948168 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77e4d840-8240-45e1-8daf-b70feb18e4b1-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:47.948348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:47.948227 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgcbx\" (UniqueName: \"kubernetes.io/projected/77e4d840-8240-45e1-8daf-b70feb18e4b1-kube-api-access-dgcbx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:48.140771 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:48.140743 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57c8275_0087_4666_b7fa_b1a8785040e9.slice/crio-e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144 WatchSource:0}: Error finding container e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144: Status 404 returned error can't find the container with id e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144 Oct 30 05:49:48.141898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.141871 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s"] Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.693428 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" event={"ID":"b57c8275-0087-4666-b7fa-b1a8785040e9","Type":"ContainerStarted","Data":"946086a43f45db72cb1159047a1f647d6ff390a14a05af583011c887ec675278"} Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.693492 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" event={"ID":"b57c8275-0087-4666-b7fa-b1a8785040e9","Type":"ContainerStarted","Data":"e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144"} Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.695259 6171 generic.go:356] "Generic (PLEG): container finished" podID="ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" containerID="9ee52fcbbbca55553b82dda3c6e506799ee43f8c7a799939c690273032dda8b9" exitCode=0 Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.695378 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" event={"ID":"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5","Type":"ContainerDied","Data":"9ee52fcbbbca55553b82dda3c6e506799ee43f8c7a799939c690273032dda8b9"} Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.697104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" event={"ID":"77e4d840-8240-45e1-8daf-b70feb18e4b1","Type":"ContainerDied","Data":"ac8faff4cea761336b6c8ac124d014f4f6b71e73e74c4f2bc5e5c481d41e6110"} Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.697138 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac8faff4cea761336b6c8ac124d014f4f6b71e73e74c4f2bc5e5c481d41e6110" Oct 30 05:49:48.697941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.697169 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s" Oct 30 05:49:48.737017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:48.736974 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" podStartSLOduration=1.7369605190000001 podStartE2EDuration="1.736960519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:48.734442671 +0000 UTC m=+1047.174819863" watchObservedRunningTime="2026-10-30 05:49:48.736960519 +0000 UTC m=+1047.177337712" Oct 30 05:49:49.932069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:49.931984 6171 generic.go:356] "Generic (PLEG): container finished" podID="b57c8275-0087-4666-b7fa-b1a8785040e9" containerID="946086a43f45db72cb1159047a1f647d6ff390a14a05af583011c887ec675278" exitCode=0 Oct 30 05:49:49.940075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:49.932653 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" event={"ID":"b57c8275-0087-4666-b7fa-b1a8785040e9","Type":"ContainerDied","Data":"946086a43f45db72cb1159047a1f647d6ff390a14a05af583011c887ec675278"} Oct 30 05:49:50.669155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.668571 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:49:50.673390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.669623 6171 cpu_manager.go:414] "containerMap: removing container" podUID="77e4d840-8240-45e1-8daf-b70feb18e4b1" containerName="revoke-keys" Oct 30 05:49:50.673390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.669645 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77e4d840-8240-45e1-8daf-b70feb18e4b1" containerName="revoke-keys" Oct 30 05:49:50.673390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.669883 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="77e4d840-8240-45e1-8daf-b70feb18e4b1" containerName="revoke-keys" Oct 30 05:49:50.695655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.691782 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:49:50.695655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.691961 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864107 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864187 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864218 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864249 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdgq8\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864280 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864428 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864491 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.904216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.864572 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.975585 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976199 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976232 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976266 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdgq8\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976301 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976352 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976470 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.976601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.976537 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.984109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.981587 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:50.984109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:50.982353 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.010461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.004394 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.042098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.018138 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.042098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.018915 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.130133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.110515 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.138883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.136812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.278236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.276629 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdgq8\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.393118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.284433 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.393118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.299750 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:51.427070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.416205 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:51.522445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.522412 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\") pod \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " Oct 30 05:49:51.522671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.522652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\" (UniqueName: \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\") pod \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " Oct 30 05:49:51.522807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.522789 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\") pod \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\" (UID: \"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5\") " Oct 30 05:49:51.524314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.524289 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca" pod "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" (UID: "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:51.526603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.526579 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:51.530221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.530193 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:49:51.531442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.531420 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" containerName="revoke-keys" Oct 30 05:49:51.531584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.531569 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" containerName="revoke-keys" Oct 30 05:49:51.531865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.531847 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" containerName="revoke-keys" Oct 30 05:49:51.545065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.543331 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.564115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.564089 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:49:51.567804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.567767 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs" pod "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" (UID: "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5"). InnerVolumeSpecName "kube-api-access-6h9fs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:51.568231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.568204 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca" pod "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" (UID: "ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:51.574784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.573164 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 05:49:51.602814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.601286 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 05:49:51.602814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.601406 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.640923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.639753 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6h9fs\" (UniqueName: \"kubernetes.io/projected/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-kube-api-access-6h9fs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:51.640923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.639785 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740674 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740733 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740779 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzfx2\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740814 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740866 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740938 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740961 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8rh\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.740988 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.741086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.741012 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.745817 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.745885 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.745948 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.745977 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.746024 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.746084 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.746222 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.746248 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.747426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.746276 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849379 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849472 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849503 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8rh\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849556 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849631 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849697 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849752 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849783 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849828 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.849870 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850006 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850031 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850077 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.850104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850110 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.851104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.851104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850224 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzfx2\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.851104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850256 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.851104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.850724 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.851104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.851033 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.879001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.871974 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.879001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.872302 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.879001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.872561 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.879001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.876670 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.879001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.878969 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.881772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.879631 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.881772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.880246 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.889020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.888611 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.889020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.888892 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.890071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.889825 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.891541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.891474 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.891679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.891613 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.896556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.896514 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.906089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.905752 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.906218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.906165 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzfx2\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:51.909027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.909002 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8rh\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:51.909143 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.909120 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:51.951171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.951143 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\") pod \"b57c8275-0087-4666-b7fa-b1a8785040e9\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " Oct 30 05:49:51.951331 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.951215 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\") pod \"b57c8275-0087-4666-b7fa-b1a8785040e9\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " Oct 30 05:49:51.951331 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.951281 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\" (UniqueName: \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\") pod \"b57c8275-0087-4666-b7fa-b1a8785040e9\" (UID: \"b57c8275-0087-4666-b7fa-b1a8785040e9\") " Oct 30 05:49:51.952387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.952347 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca" pod "b57c8275-0087-4666-b7fa-b1a8785040e9" (UID: "b57c8275-0087-4666-b7fa-b1a8785040e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:51.954145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.954109 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4" pod "b57c8275-0087-4666-b7fa-b1a8785040e9" (UID: "b57c8275-0087-4666-b7fa-b1a8785040e9"). InnerVolumeSpecName "kube-api-access-ddlf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:51.954674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.954656 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca" pod "b57c8275-0087-4666-b7fa-b1a8785040e9" (UID: "b57c8275-0087-4666-b7fa-b1a8785040e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:51.976896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:51.976875 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:52.030404 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:52.030360 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987e8c69_d497_4b8f_b727_21ac43bb7bb4.slice/crio-4fa9880f6d40d2f83d29146b509698fac710ee0cff8ee75ed83355f90875baf1 WatchSource:0}: Error finding container 4fa9880f6d40d2f83d29146b509698fac710ee0cff8ee75ed83355f90875baf1: Status 404 returned error can't find the container with id 4fa9880f6d40d2f83d29146b509698fac710ee0cff8ee75ed83355f90875baf1 Oct 30 05:49:52.032295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.032266 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:49:52.034416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.034380 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.034502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.034463 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.034542 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.034510 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.053811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.053785 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:52.054134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.053815 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b57c8275-0087-4666-b7fa-b1a8785040e9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:52.054134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.053824 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddlf4\" (UniqueName: \"kubernetes.io/projected/b57c8275-0087-4666-b7fa-b1a8785040e9-kube-api-access-ddlf4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:49:52.208865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.207364 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:52.530482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.530456 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" Oct 30 05:49:52.530609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.530555 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx" event={"ID":"ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5","Type":"ContainerDied","Data":"e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b"} Oct 30 05:49:52.530609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.530604 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e48f287d3b4e4a7a61d22237a4a8d546caa09e2b511d265351b0efe8d7824f0b" Oct 30 05:49:52.533608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.533581 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 05:49:52.536865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.536834 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.536965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.536907 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.536965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.536949 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.537206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.537184 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" event={"ID":"b57c8275-0087-4666-b7fa-b1a8785040e9","Type":"ContainerDied","Data":"e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144"} Oct 30 05:49:52.537276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.537225 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5283aa3c4abc486a95118e5c594ddf7b54b9d6411d701238110d245118ec144" Oct 30 05:49:52.537343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.537292 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s" Oct 30 05:49:52.540484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.540447 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" event={"ID":"987e8c69-d497-4b8f-b727-21ac43bb7bb4","Type":"ContainerStarted","Data":"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945"} Oct 30 05:49:52.540582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.540498 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" event={"ID":"987e8c69-d497-4b8f-b727-21ac43bb7bb4","Type":"ContainerStarted","Data":"4fa9880f6d40d2f83d29146b509698fac710ee0cff8ee75ed83355f90875baf1"} Oct 30 05:49:52.594210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.594162 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podStartSLOduration=2.594148348 podStartE2EDuration="2.594148348s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:52.587475292 +0000 UTC m=+1051.027852486" watchObservedRunningTime="2026-10-30 05:49:52.594148348 +0000 UTC m=+1051.034525541" Oct 30 05:49:52.633915 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:52.633888 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cca739a_b888_416c_98fc_64457cb99655.slice/crio-1e0df3be34aab6f4e92a9fc0705dfddc499c5b6020889d64f5689c083da213b9 WatchSource:0}: Error finding container 1e0df3be34aab6f4e92a9fc0705dfddc499c5b6020889d64f5689c083da213b9: Status 404 returned error can't find the container with id 1e0df3be34aab6f4e92a9fc0705dfddc499c5b6020889d64f5689c083da213b9 Oct 30 05:49:52.636356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.636329 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:49:52.636830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.636798 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.636908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.636872 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:52.636979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:52.636919 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:49:53.309581 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.309550 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:49:53.310218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.310132 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:53.310373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.310354 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:53.573702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.573592 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" event={"ID":"f644c123-ad3b-47d6-a80c-3b160bd5edcc","Type":"ContainerStarted","Data":"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1"} Oct 30 05:49:53.575133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.575108 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" event={"ID":"f644c123-ad3b-47d6-a80c-3b160bd5edcc","Type":"ContainerStarted","Data":"35aae412849d81582a59debecd5fa8bab31f80e7294012a368a56c689a4d6625"} Oct 30 05:49:53.578692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.578670 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" event={"ID":"2cca739a-b888-416c-98fc-64457cb99655","Type":"ContainerStarted","Data":"a71d904634d3b3d8fa8091b5f29f0a9dabc76af4264ae4e8b2e8842fe8c164f3"} Oct 30 05:49:53.579252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.579235 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" event={"ID":"2cca739a-b888-416c-98fc-64457cb99655","Type":"ContainerStarted","Data":"1e0df3be34aab6f4e92a9fc0705dfddc499c5b6020889d64f5689c083da213b9"} Oct 30 05:49:53.728068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.727831 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podStartSLOduration=2.727812144 podStartE2EDuration="2.727812144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:53.664482477 +0000 UTC m=+1052.104859671" watchObservedRunningTime="2026-10-30 05:49:53.727812144 +0000 UTC m=+1052.168189337" Oct 30 05:49:53.979763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.979731 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:49:53.998341 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.998307 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:53.998627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:53.998585 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:54.232733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.221798 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:49:54.246070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.239164 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.246070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.239227 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:54.315074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.314493 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.315074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.314554 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:54.978276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.978246 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:54.978451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:54.978294 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:55.225448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.225391 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.225659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.225458 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:55.306127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.305375 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.306127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.305438 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:55.977678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.977627 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:55.978063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:55.977701 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:56.221071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.220468 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.221071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.220532 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:56.312128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.308402 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.312128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.308473 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:56.490069 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:49:56.485410 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:49:56.977759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.977719 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:56.978182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:56.977776 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:57.223213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.219621 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.223213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.219681 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:57.309127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.308445 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.309127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.308506 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:57.873694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.873641 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podStartSLOduration=6.873628267 podStartE2EDuration="6.873628267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:53.726421844 +0000 UTC m=+1052.166799038" watchObservedRunningTime="2026-10-30 05:49:57.873628267 +0000 UTC m=+1056.314005460" Oct 30 05:49:57.877549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.877527 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf"] Oct 30 05:49:57.878198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.878185 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b57c8275-0087-4666-b7fa-b1a8785040e9" containerName="revoke-keys" Oct 30 05:49:57.878237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.878200 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57c8275-0087-4666-b7fa-b1a8785040e9" containerName="revoke-keys" Oct 30 05:49:57.878340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.878331 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57c8275-0087-4666-b7fa-b1a8785040e9" containerName="revoke-keys" Oct 30 05:49:57.889575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.889550 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:57.922393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.922360 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:57.923258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.923235 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:49:57.923443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.923430 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:57.923532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.923466 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:57.970597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.969130 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf"] Oct 30 05:49:57.977479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.977445 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:57.977612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:57.977521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:58.066717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.066672 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.067270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.066787 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.067270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.066957 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7shqx\" (UniqueName: \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.168684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.168594 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.168684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.168664 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.168921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.168764 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7shqx\" (UniqueName: \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.169402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.169373 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.171413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.171393 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.216070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.215030 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.216070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.215124 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:58.229095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.221983 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7shqx\" (UniqueName: \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\") pod \"maas-api-revoke-sub-022c89b8-dn9jf\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.278668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.278640 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:49:58.311084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.308913 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.311084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.309082 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:58.978065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.978018 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" start-of-body= Oct 30 05:49:58.978222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:58.978095 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": dial tcp 10.133.0.109:15021: connect: connection refused" Oct 30 05:49:59.159995 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:49:59.159942 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccf2ed18_fd10_409d_9e6a_375fe2a373f5.slice/crio-3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4 WatchSource:0}: Error finding container 3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4: Status 404 returned error can't find the container with id 3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4 Oct 30 05:49:59.161024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.160998 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf"] Oct 30 05:49:59.218074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.216984 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:49:59.218074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.217094 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:49:59.310070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.308468 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" start-of-body= Oct 30 05:49:59.310070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.308540 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": dial tcp 10.133.0.107:15021: connect: connection refused" Oct 30 05:49:59.953127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.949834 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" event={"ID":"ccf2ed18-fd10-409d-9e6a-375fe2a373f5","Type":"ContainerStarted","Data":"3324a174d57fbd844a803a998974f393a47039b8c585926489cc149be3a47c9e"} Oct 30 05:49:59.953127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.949901 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" event={"ID":"ccf2ed18-fd10-409d-9e6a-375fe2a373f5","Type":"ContainerStarted","Data":"3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4"} Oct 30 05:49:59.992067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:49:59.991633 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" podStartSLOduration=2.991614938 podStartE2EDuration="2.991614938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:59.990443882 +0000 UTC m=+1058.430821075" watchObservedRunningTime="2026-10-30 05:49:59.991614938 +0000 UTC m=+1058.431992133" Oct 30 05:50:00.180306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.180271 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:50:00.186582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.182191 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:50:00.186582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.182892 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 05:50:00.212160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.212030 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body= Oct 30 05:50:00.212160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.212129 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" Oct 30 05:50:00.405924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.404969 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:50:00.430662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.430566 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:50:00.432386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.432355 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:50:00.958901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.958872 6171 generic.go:356] "Generic (PLEG): container finished" podID="ccf2ed18-fd10-409d-9e6a-375fe2a373f5" containerID="3324a174d57fbd844a803a998974f393a47039b8c585926489cc149be3a47c9e" exitCode=0 Oct 30 05:50:00.959073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:00.958964 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" event={"ID":"ccf2ed18-fd10-409d-9e6a-375fe2a373f5","Type":"ContainerDied","Data":"3324a174d57fbd844a803a998974f393a47039b8c585926489cc149be3a47c9e"} Oct 30 05:50:01.614964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.614933 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:50:01.615500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.615479 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:50:01.616438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.616408 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:50:01.682784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.682750 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp"] Oct 30 05:50:01.687652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.687627 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.701683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.701647 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp"] Oct 30 05:50:01.752065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.751343 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csqtx\" (UniqueName: \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.752065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.751445 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.752065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.751516 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.852870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.852840 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.853064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.852959 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csqtx\" (UniqueName: \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.853160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.853080 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.853648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.853628 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.855449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.855418 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:01.883619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:01.883560 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csqtx\" (UniqueName: \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\") pod \"maas-api-revoke-sub-763e519d-dmntp\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:02.029068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.022315 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:02.340215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.340184 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:50:02.439704 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.439630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\") pod \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " Oct 30 05:50:02.439704 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.439668 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\" (UniqueName: \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\") pod \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " Oct 30 05:50:02.439961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.439940 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\") pod \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\" (UID: \"ccf2ed18-fd10-409d-9e6a-375fe2a373f5\") " Oct 30 05:50:02.440290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.440264 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca" pod "ccf2ed18-fd10-409d-9e6a-375fe2a373f5" (UID: "ccf2ed18-fd10-409d-9e6a-375fe2a373f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:02.440655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.440637 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:02.441984 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.441960 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca" pod "ccf2ed18-fd10-409d-9e6a-375fe2a373f5" (UID: "ccf2ed18-fd10-409d-9e6a-375fe2a373f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:02.442096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.442007 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx" pod "ccf2ed18-fd10-409d-9e6a-375fe2a373f5" (UID: "ccf2ed18-fd10-409d-9e6a-375fe2a373f5"). InnerVolumeSpecName "kube-api-access-7shqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:02.532900 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:02.532853 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda639fda0_ccc6_4e1d_b653_400094e86ea8.slice/crio-9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f WatchSource:0}: Error finding container 9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f: Status 404 returned error can't find the container with id 9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f Oct 30 05:50:02.534796 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.534773 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp"] Oct 30 05:50:02.542200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.542182 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:02.542310 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.542203 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7shqx\" (UniqueName: \"kubernetes.io/projected/ccf2ed18-fd10-409d-9e6a-375fe2a373f5-kube-api-access-7shqx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:02.977036 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.977001 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" event={"ID":"a639fda0-ccc6-4e1d-b653-400094e86ea8","Type":"ContainerStarted","Data":"8ca0dc36f616dbc5510500d700ea923575a3a7faf2b3d9f1cfb1d524c4120bd9"} Oct 30 05:50:02.977503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.977084 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" event={"ID":"a639fda0-ccc6-4e1d-b653-400094e86ea8","Type":"ContainerStarted","Data":"9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f"} Oct 30 05:50:02.978620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.978600 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" Oct 30 05:50:02.978721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.978596 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf" event={"ID":"ccf2ed18-fd10-409d-9e6a-375fe2a373f5","Type":"ContainerDied","Data":"3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4"} Oct 30 05:50:02.978721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:02.978718 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3cd504f0368c3b6cbb3b65420dfe130d7daff7301619ad943f5d1d642b2994a4" Oct 30 05:50:03.025111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:03.023748 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" podStartSLOduration=2.023715569 podStartE2EDuration="2.023715569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:03.016110357 +0000 UTC m=+1061.456487548" watchObservedRunningTime="2026-10-30 05:50:03.023715569 +0000 UTC m=+1061.464092762" Oct 30 05:50:03.989729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:03.989696 6171 generic.go:356] "Generic (PLEG): container finished" podID="a639fda0-ccc6-4e1d-b653-400094e86ea8" containerID="8ca0dc36f616dbc5510500d700ea923575a3a7faf2b3d9f1cfb1d524c4120bd9" exitCode=0 Oct 30 05:50:03.990161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:03.989762 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" event={"ID":"a639fda0-ccc6-4e1d-b653-400094e86ea8","Type":"ContainerDied","Data":"8ca0dc36f616dbc5510500d700ea923575a3a7faf2b3d9f1cfb1d524c4120bd9"} Oct 30 05:50:05.158637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.158607 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:05.294378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.294291 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\") pod \"a639fda0-ccc6-4e1d-b653-400094e86ea8\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " Oct 30 05:50:05.294593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.294578 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\" (UniqueName: \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\") pod \"a639fda0-ccc6-4e1d-b653-400094e86ea8\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " Oct 30 05:50:05.294674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.294604 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\") pod \"a639fda0-ccc6-4e1d-b653-400094e86ea8\" (UID: \"a639fda0-ccc6-4e1d-b653-400094e86ea8\") " Oct 30 05:50:05.294674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.294647 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca" pod "a639fda0-ccc6-4e1d-b653-400094e86ea8" (UID: "a639fda0-ccc6-4e1d-b653-400094e86ea8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:05.295144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.295122 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:05.297204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.297177 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca" pod "a639fda0-ccc6-4e1d-b653-400094e86ea8" (UID: "a639fda0-ccc6-4e1d-b653-400094e86ea8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:05.297311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.297283 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx" pod "a639fda0-ccc6-4e1d-b653-400094e86ea8" (UID: "a639fda0-ccc6-4e1d-b653-400094e86ea8"). InnerVolumeSpecName "kube-api-access-csqtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:05.396153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.396116 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csqtx\" (UniqueName: \"kubernetes.io/projected/a639fda0-ccc6-4e1d-b653-400094e86ea8-kube-api-access-csqtx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:05.396153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:05.396151 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a639fda0-ccc6-4e1d-b653-400094e86ea8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:06.008871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.008836 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" Oct 30 05:50:06.009039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.008867 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp" event={"ID":"a639fda0-ccc6-4e1d-b653-400094e86ea8","Type":"ContainerDied","Data":"9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f"} Oct 30 05:50:06.009039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.008916 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c8eb03302ce4689d336f8cd4135076ceab32a04048f7cb61a6a6fc08276652f" Oct 30 05:50:06.455818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.455734 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54"] Oct 30 05:50:06.456514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456493 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a639fda0-ccc6-4e1d-b653-400094e86ea8" containerName="revoke-keys" Oct 30 05:50:06.456514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456511 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a639fda0-ccc6-4e1d-b653-400094e86ea8" containerName="revoke-keys" Oct 30 05:50:06.456680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456536 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ccf2ed18-fd10-409d-9e6a-375fe2a373f5" containerName="revoke-keys" Oct 30 05:50:06.456680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456541 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccf2ed18-fd10-409d-9e6a-375fe2a373f5" containerName="revoke-keys" Oct 30 05:50:06.456680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456646 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a639fda0-ccc6-4e1d-b653-400094e86ea8" containerName="revoke-keys" Oct 30 05:50:06.456680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.456659 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccf2ed18-fd10-409d-9e6a-375fe2a373f5" containerName="revoke-keys" Oct 30 05:50:06.460705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.460688 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.479814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.479786 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54"] Oct 30 05:50:06.486235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.486197 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:06.486354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.486309 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:06.513077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.512998 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:06.513432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.513405 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.513591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.513570 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jw8gx\" (UniqueName: \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.513847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.513828 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.514297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.514244 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:50:06.615368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.615339 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.615514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.615398 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.615514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.615475 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jw8gx\" (UniqueName: \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.616092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.616075 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.617851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.617837 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.638580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.638559 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jw8gx\" (UniqueName: \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\") pod \"maas-api-revoke-sub-2b915f7f-vnj54\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:06.775158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:06.775130 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:07.085580 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:07.085546 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadaadbad_7a54_49cb_9929_58c80d656db3.slice/crio-d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4 WatchSource:0}: Error finding container d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4: Status 404 returned error can't find the container with id d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4 Oct 30 05:50:07.086668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:07.086531 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54"] Oct 30 05:50:08.028739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.028713 6171 generic.go:356] "Generic (PLEG): container finished" podID="adaadbad-7a54-49cb-9929-58c80d656db3" containerID="30ab555e6a959d064fab66f844699afbf9804d0a1f8366980bcf5cc7278d7d9b" exitCode=0 Oct 30 05:50:08.029114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.028759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" event={"ID":"adaadbad-7a54-49cb-9929-58c80d656db3","Type":"ContainerDied","Data":"30ab555e6a959d064fab66f844699afbf9804d0a1f8366980bcf5cc7278d7d9b"} Oct 30 05:50:08.029114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.028822 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" event={"ID":"adaadbad-7a54-49cb-9929-58c80d656db3","Type":"ContainerStarted","Data":"d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4"} Oct 30 05:50:08.134707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.134670 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl"] Oct 30 05:50:08.139202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.139184 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.158960 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.158935 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl"] Oct 30 05:50:08.244583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.244545 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.244778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.244598 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.244778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.244699 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgjwt\" (UniqueName: \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.346594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.346501 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.346594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.346556 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.346827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.346613 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgjwt\" (UniqueName: \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.347309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.347287 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.348972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.348952 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.370035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.370007 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgjwt\" (UniqueName: \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\") pod \"maas-api-revoke-sub-1b14588b-f69sl\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.452745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.452709 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:08.761763 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:08.761728 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod448e29c0_d09d_4dde_9220_e5b04d1c1707.slice/crio-c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c WatchSource:0}: Error finding container c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c: Status 404 returned error can't find the container with id c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c Oct 30 05:50:08.763503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:08.763480 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl"] Oct 30 05:50:09.042446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.041302 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" event={"ID":"448e29c0-d09d-4dde-9220-e5b04d1c1707","Type":"ContainerStarted","Data":"c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c"} Oct 30 05:50:09.212672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.212643 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:09.372488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.372395 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\" (UniqueName: \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\") pod \"adaadbad-7a54-49cb-9929-58c80d656db3\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " Oct 30 05:50:09.372657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.372610 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\") pod \"adaadbad-7a54-49cb-9929-58c80d656db3\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " Oct 30 05:50:09.372792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.372774 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\") pod \"adaadbad-7a54-49cb-9929-58c80d656db3\" (UID: \"adaadbad-7a54-49cb-9929-58c80d656db3\") " Oct 30 05:50:09.373157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.373135 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca" pod "adaadbad-7a54-49cb-9929-58c80d656db3" (UID: "adaadbad-7a54-49cb-9929-58c80d656db3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:09.373361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.373344 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.374546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.374529 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx" pod "adaadbad-7a54-49cb-9929-58c80d656db3" (UID: "adaadbad-7a54-49cb-9929-58c80d656db3"). InnerVolumeSpecName "kube-api-access-jw8gx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:09.374633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.374620 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca" pod "adaadbad-7a54-49cb-9929-58c80d656db3" (UID: "adaadbad-7a54-49cb-9929-58c80d656db3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:09.474627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.474600 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jw8gx\" (UniqueName: \"kubernetes.io/projected/adaadbad-7a54-49cb-9929-58c80d656db3-kube-api-access-jw8gx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.474627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:09.474625 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/adaadbad-7a54-49cb-9929-58c80d656db3-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:10.052736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:10.052701 6171 generic.go:356] "Generic (PLEG): container finished" podID="448e29c0-d09d-4dde-9220-e5b04d1c1707" containerID="f2c4a87f236c59680bffda9223517157b0a8c58d324ad09b55371e324e95cf6a" exitCode=0 Oct 30 05:50:10.053173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:10.052789 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" event={"ID":"448e29c0-d09d-4dde-9220-e5b04d1c1707","Type":"ContainerDied","Data":"f2c4a87f236c59680bffda9223517157b0a8c58d324ad09b55371e324e95cf6a"} Oct 30 05:50:10.054332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:10.054314 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" Oct 30 05:50:10.054455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:10.054359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54" event={"ID":"adaadbad-7a54-49cb-9929-58c80d656db3","Type":"ContainerDied","Data":"d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4"} Oct 30 05:50:10.054455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:10.054399 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d64c52a79049fc22e0289c6249eabe510b7ee5c258dc0af4c484d2d33b5be9f4" Oct 30 05:50:11.236278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.236255 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:11.404646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.404571 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\") pod \"448e29c0-d09d-4dde-9220-e5b04d1c1707\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " Oct 30 05:50:11.404782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.404676 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\" (UniqueName: \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\") pod \"448e29c0-d09d-4dde-9220-e5b04d1c1707\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " Oct 30 05:50:11.404829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.404817 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\") pod \"448e29c0-d09d-4dde-9220-e5b04d1c1707\" (UID: \"448e29c0-d09d-4dde-9220-e5b04d1c1707\") " Oct 30 05:50:11.405165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.405136 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca" pod "448e29c0-d09d-4dde-9220-e5b04d1c1707" (UID: "448e29c0-d09d-4dde-9220-e5b04d1c1707"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:11.405494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.405474 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:11.407146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.407113 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca" pod "448e29c0-d09d-4dde-9220-e5b04d1c1707" (UID: "448e29c0-d09d-4dde-9220-e5b04d1c1707"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:11.407241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.407192 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt" pod "448e29c0-d09d-4dde-9220-e5b04d1c1707" (UID: "448e29c0-d09d-4dde-9220-e5b04d1c1707"). InnerVolumeSpecName "kube-api-access-bgjwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:11.517068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.516719 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/448e29c0-d09d-4dde-9220-e5b04d1c1707-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:11.517068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:11.516758 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgjwt\" (UniqueName: \"kubernetes.io/projected/448e29c0-d09d-4dde-9220-e5b04d1c1707-kube-api-access-bgjwt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:12.107120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:12.106464 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" event={"ID":"448e29c0-d09d-4dde-9220-e5b04d1c1707","Type":"ContainerDied","Data":"c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c"} Oct 30 05:50:12.107120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:12.106549 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c95ad47785c3d3a575c63d8db4f5523a12c6442a8d5797836df381f7bc763c7c" Oct 30 05:50:12.107120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:12.106640 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl" Oct 30 05:50:13.044187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:13.026732 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.121254 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:13.121195 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.384382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:13.384283 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:14.036260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.036205 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:50:14.051115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.037248 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:14.082133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.080510 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.392696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.392589 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.427074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.393846 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:50:14.588080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:14.521932 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.108122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.107384 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.108904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.108877 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:50:15.126153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.117153 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.126153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.118123 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:50:15.397255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.397155 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:15.433072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.430786 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.494823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.494776 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.592239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.592197 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.602741 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.602713 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:50:15.874165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:15.845265 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.125123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.122161 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.125730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.125674 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.135473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.129266 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.162692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.146357 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:50:16.274089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.259647 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:50:16.287577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.281866 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:16.287577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.283288 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.324020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.323575 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:50:16.348900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.327419 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:16.348900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.327488 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.407858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.406251 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:16.440957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.437133 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:50:16.440957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.437840 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:50:16.510593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.510557 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:16.511153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:16.510839 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:17.166082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.166006 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:50:17.177491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.167937 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:17.207416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.207371 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:50:17.549267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.549198 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:50:17.562688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.562661 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:17.568788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.568765 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:50:17.569389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:17.569369 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:50:18.785925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.785898 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj"] Oct 30 05:50:18.786585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786570 6171 cpu_manager.go:414] "containerMap: removing container" podUID="448e29c0-d09d-4dde-9220-e5b04d1c1707" containerName="revoke-keys" Oct 30 05:50:18.786638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786586 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="448e29c0-d09d-4dde-9220-e5b04d1c1707" containerName="revoke-keys" Oct 30 05:50:18.786638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786624 6171 cpu_manager.go:414] "containerMap: removing container" podUID="adaadbad-7a54-49cb-9929-58c80d656db3" containerName="revoke-keys" Oct 30 05:50:18.786638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786629 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="adaadbad-7a54-49cb-9929-58c80d656db3" containerName="revoke-keys" Oct 30 05:50:18.786732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786712 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="448e29c0-d09d-4dde-9220-e5b04d1c1707" containerName="revoke-keys" Oct 30 05:50:18.786732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.786729 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="adaadbad-7a54-49cb-9929-58c80d656db3" containerName="revoke-keys" Oct 30 05:50:18.790475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.790459 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:18.805396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.805369 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:18.805508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.805420 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:50:18.809713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.809689 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj"] Oct 30 05:50:18.815805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.815779 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:18.815932 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.815909 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:18.892687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.892656 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxdf\" (UniqueName: \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:18.892844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.892699 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:18.892889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:18.892843 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.007678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.006535 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxdf\" (UniqueName: \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.007678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.006609 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.007678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.006787 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.020136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.018236 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.020758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.020701 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.046888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.046807 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxdf\" (UniqueName: \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\") pod \"maas-api-revoke-sub-9b23add5-ms2rj\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.105535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.105505 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:19.115465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.115443 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66"] Oct 30 05:50:19.121425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.121401 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.143064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.142985 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66"] Oct 30 05:50:19.209868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.209833 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.209981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.209900 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.209981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.209939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htkjn\" (UniqueName: \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.311491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.311412 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.311491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.311468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.311491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.311492 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htkjn\" (UniqueName: \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.312148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.312115 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.313973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.313949 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.334537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.334516 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htkjn\" (UniqueName: \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\") pod \"maas-api-revoke-sub-d3805d87-9vc66\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.431864 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:19.431835 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78d7095c_7690_4c21_8657_f107d417f34d.slice/crio-d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113 WatchSource:0}: Error finding container d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113: Status 404 returned error can't find the container with id d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113 Oct 30 05:50:19.433787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.433767 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj"] Oct 30 05:50:19.463200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.463180 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:19.558079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.558037 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" event={"ID":"78d7095c-7690-4c21-8657-f107d417f34d","Type":"ContainerStarted","Data":"d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113"} Oct 30 05:50:19.770974 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:19.770948 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93317926_5246_42e4_b051_0fa3e16d0827.slice/crio-17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1 WatchSource:0}: Error finding container 17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1: Status 404 returned error can't find the container with id 17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1 Oct 30 05:50:19.772125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:19.772097 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66"] Oct 30 05:50:20.569501 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:20.569421 6171 generic.go:356] "Generic (PLEG): container finished" podID="78d7095c-7690-4c21-8657-f107d417f34d" containerID="cbfc96a29f9c8fc9d4837747ce320cfb6b5ae88754c8788d1449e5f44823ecb2" exitCode=0 Oct 30 05:50:20.569934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:20.569505 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" event={"ID":"78d7095c-7690-4c21-8657-f107d417f34d","Type":"ContainerDied","Data":"cbfc96a29f9c8fc9d4837747ce320cfb6b5ae88754c8788d1449e5f44823ecb2"} Oct 30 05:50:20.571520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:20.571469 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" event={"ID":"93317926-5246-42e4-b051-0fa3e16d0827","Type":"ContainerStarted","Data":"a678e875e8de3b941566e900205df5f6b99cefa06bd98968b1dc1f2af12dead4"} Oct 30 05:50:20.571665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:20.571525 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" event={"ID":"93317926-5246-42e4-b051-0fa3e16d0827","Type":"ContainerStarted","Data":"17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1"} Oct 30 05:50:20.660556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:20.660499 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" podStartSLOduration=1.660486106 podStartE2EDuration="1.660486106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:20.655538083 +0000 UTC m=+1079.095915275" watchObservedRunningTime="2026-10-30 05:50:20.660486106 +0000 UTC m=+1079.100863299" Oct 30 05:50:21.098176 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.098138 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:50:21.103344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.103320 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.138382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.138355 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:50:21.245677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245644 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.245842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245685 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.245842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmm2\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.245842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.245972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245871 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.245972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.245952 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.246097 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.246027 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.246150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.246136 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.246187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.246181 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347281 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347369 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347414 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347508 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347622 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347630 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347702 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmm2\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347756 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.347915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347891 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.348288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.347939 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.348288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.348081 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.348464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.348283 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.348464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.348432 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.349830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.349810 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.349961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.349943 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.377566 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.377517 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmm2\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.377566 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.377537 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.427099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.427077 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:21.581924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.581895 6171 generic.go:356] "Generic (PLEG): container finished" podID="93317926-5246-42e4-b051-0fa3e16d0827" containerID="a678e875e8de3b941566e900205df5f6b99cefa06bd98968b1dc1f2af12dead4" exitCode=0 Oct 30 05:50:21.582401 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.581978 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" event={"ID":"93317926-5246-42e4-b051-0fa3e16d0827","Type":"ContainerDied","Data":"a678e875e8de3b941566e900205df5f6b99cefa06bd98968b1dc1f2af12dead4"} Oct 30 05:50:21.740007 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.739980 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:21.759647 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:21.759621 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41b9538b_7e10_49ba_9cd4_d2e188f2742a.slice/crio-f0cbb13f2e6bf1784bf48ba4d2d64247f1042f43813a666f0a26e3320779c680 WatchSource:0}: Error finding container f0cbb13f2e6bf1784bf48ba4d2d64247f1042f43813a666f0a26e3320779c680: Status 404 returned error can't find the container with id f0cbb13f2e6bf1784bf48ba4d2d64247f1042f43813a666f0a26e3320779c680 Oct 30 05:50:21.761201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.761166 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:50:21.762587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.762558 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:21.762661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.762616 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:21.762661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.762647 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:21.857431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.857404 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\") pod \"78d7095c-7690-4c21-8657-f107d417f34d\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " Oct 30 05:50:21.857558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.857466 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\" (UniqueName: \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\") pod \"78d7095c-7690-4c21-8657-f107d417f34d\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " Oct 30 05:50:21.857558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.857497 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\") pod \"78d7095c-7690-4c21-8657-f107d417f34d\" (UID: \"78d7095c-7690-4c21-8657-f107d417f34d\") " Oct 30 05:50:21.857804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.857772 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca" pod "78d7095c-7690-4c21-8657-f107d417f34d" (UID: "78d7095c-7690-4c21-8657-f107d417f34d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:21.858146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.858110 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:21.859618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.859595 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca" pod "78d7095c-7690-4c21-8657-f107d417f34d" (UID: "78d7095c-7690-4c21-8657-f107d417f34d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:21.859776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.859761 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf" pod "78d7095c-7690-4c21-8657-f107d417f34d" (UID: "78d7095c-7690-4c21-8657-f107d417f34d"). InnerVolumeSpecName "kube-api-access-6sxdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:21.959617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.959551 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sxdf\" (UniqueName: \"kubernetes.io/projected/78d7095c-7690-4c21-8657-f107d417f34d-kube-api-access-6sxdf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:21.959617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:21.959579 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78d7095c-7690-4c21-8657-f107d417f34d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:22.108723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.108693 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn"] Oct 30 05:50:22.109406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.109391 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78d7095c-7690-4c21-8657-f107d417f34d" containerName="revoke-keys" Oct 30 05:50:22.109454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.109407 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d7095c-7690-4c21-8657-f107d417f34d" containerName="revoke-keys" Oct 30 05:50:22.109546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.109537 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d7095c-7690-4c21-8657-f107d417f34d" containerName="revoke-keys" Oct 30 05:50:22.113757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.113736 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.136312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.136285 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn"] Oct 30 05:50:22.265141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.264616 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.265141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.264670 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4gvc\" (UniqueName: \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.265141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.264832 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.366666 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.366632 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.366797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.366670 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4gvc\" (UniqueName: \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.366797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.366781 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.367495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.367465 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.369386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.369363 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.406070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.403686 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4gvc\" (UniqueName: \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\") pod \"maas-api-revoke-sub-4d689555-rtgqn\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.427902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.427874 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:22.735070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.734572 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" Oct 30 05:50:22.740938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.732444 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" event={"ID":"41b9538b-7e10-49ba-9cd4-d2e188f2742a","Type":"ContainerStarted","Data":"538cd677d55f215392c79262a74a13372260e97932c40e56c623f3f8fc563c52"} Oct 30 05:50:22.741127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.740970 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" event={"ID":"41b9538b-7e10-49ba-9cd4-d2e188f2742a","Type":"ContainerStarted","Data":"f0cbb13f2e6bf1784bf48ba4d2d64247f1042f43813a666f0a26e3320779c680"} Oct 30 05:50:22.741127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.740998 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj" event={"ID":"78d7095c-7690-4c21-8657-f107d417f34d","Type":"ContainerDied","Data":"d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113"} Oct 30 05:50:22.741127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.741025 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d82bb385ea807424be5f029104a9551c02c2c0c23940f437a712dd8caccd0113" Oct 30 05:50:22.813507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.812931 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podStartSLOduration=1.8129157930000002 podStartE2EDuration="1.812915793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:22.784553233 +0000 UTC m=+1081.224930426" watchObservedRunningTime="2026-10-30 05:50:22.812915793 +0000 UTC m=+1081.253292988" Oct 30 05:50:22.925494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.925470 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:22.964459 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:22.964429 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00725e0e_3a03_4e20_8aa3_50c733c5159d.slice/crio-e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8 WatchSource:0}: Error finding container e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8: Status 404 returned error can't find the container with id e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8 Oct 30 05:50:22.967139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:22.967117 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn"] Oct 30 05:50:23.118305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.118214 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\") pod \"93317926-5246-42e4-b051-0fa3e16d0827\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " Oct 30 05:50:23.118483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.118334 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\") pod \"93317926-5246-42e4-b051-0fa3e16d0827\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " Oct 30 05:50:23.118551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.118479 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\" (UniqueName: \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\") pod \"93317926-5246-42e4-b051-0fa3e16d0827\" (UID: \"93317926-5246-42e4-b051-0fa3e16d0827\") " Oct 30 05:50:23.118607 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.118556 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca" pod "93317926-5246-42e4-b051-0fa3e16d0827" (UID: "93317926-5246-42e4-b051-0fa3e16d0827"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:23.119006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.118987 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:23.211501 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.211462 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn" pod "93317926-5246-42e4-b051-0fa3e16d0827" (UID: "93317926-5246-42e4-b051-0fa3e16d0827"). InnerVolumeSpecName "kube-api-access-htkjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:23.211616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.211587 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca" pod "93317926-5246-42e4-b051-0fa3e16d0827" (UID: "93317926-5246-42e4-b051-0fa3e16d0827"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:23.220677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.220652 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93317926-5246-42e4-b051-0fa3e16d0827-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:23.220677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.220678 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htkjn\" (UniqueName: \"kubernetes.io/projected/93317926-5246-42e4-b051-0fa3e16d0827-kube-api-access-htkjn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:23.427405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.427378 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:23.428780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.428760 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:50:23.428864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.428802 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 05:50:23.743756 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.743725 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" Oct 30 05:50:23.744183 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.743758 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66" event={"ID":"93317926-5246-42e4-b051-0fa3e16d0827","Type":"ContainerDied","Data":"17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1"} Oct 30 05:50:23.744183 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.743805 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17c879e4e95f873f01bc4e7371d3d2e57c343d6dbc67de33b5aaa866557196a1" Oct 30 05:50:23.745751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.745653 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" event={"ID":"00725e0e-3a03-4e20-8aa3-50c733c5159d","Type":"ContainerStarted","Data":"50533096d698dcf73ede09f2bfbff290c9f4515311d1f8f60185db56df177251"} Oct 30 05:50:23.745893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.745726 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" event={"ID":"00725e0e-3a03-4e20-8aa3-50c733c5159d","Type":"ContainerStarted","Data":"e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8"} Oct 30 05:50:23.803070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:23.798573 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" podStartSLOduration=1.798553861 podStartE2EDuration="1.798553861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:23.797385558 +0000 UTC m=+1082.237762752" watchObservedRunningTime="2026-10-30 05:50:23.798553861 +0000 UTC m=+1082.238931058" Oct 30 05:50:24.427792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:24.427703 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:50:24.427792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:24.427767 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 05:50:24.755746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:24.755711 6171 generic.go:356] "Generic (PLEG): container finished" podID="00725e0e-3a03-4e20-8aa3-50c733c5159d" containerID="50533096d698dcf73ede09f2bfbff290c9f4515311d1f8f60185db56df177251" exitCode=0 Oct 30 05:50:24.756141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:24.755797 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" event={"ID":"00725e0e-3a03-4e20-8aa3-50c733c5159d","Type":"ContainerDied","Data":"50533096d698dcf73ede09f2bfbff290c9f4515311d1f8f60185db56df177251"} Oct 30 05:50:25.427835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.427804 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:50:25.428005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.427854 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 05:50:25.953358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.953331 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:25.964458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.964437 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\") pod \"00725e0e-3a03-4e20-8aa3-50c733c5159d\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " Oct 30 05:50:25.964828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.964809 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\") pod \"00725e0e-3a03-4e20-8aa3-50c733c5159d\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " Oct 30 05:50:25.965025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.965009 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\" (UniqueName: \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\") pod \"00725e0e-3a03-4e20-8aa3-50c733c5159d\" (UID: \"00725e0e-3a03-4e20-8aa3-50c733c5159d\") " Oct 30 05:50:25.965398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.965371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca" pod "00725e0e-3a03-4e20-8aa3-50c733c5159d" (UID: "00725e0e-3a03-4e20-8aa3-50c733c5159d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:25.966335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.966313 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:25.966686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.966667 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca" pod "00725e0e-3a03-4e20-8aa3-50c733c5159d" (UID: "00725e0e-3a03-4e20-8aa3-50c733c5159d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:25.968415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:25.968371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc" pod "00725e0e-3a03-4e20-8aa3-50c733c5159d" (UID: "00725e0e-3a03-4e20-8aa3-50c733c5159d"). InnerVolumeSpecName "kube-api-access-z4gvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:26.067913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.067842 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/00725e0e-3a03-4e20-8aa3-50c733c5159d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:26.068130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.068116 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4gvc\" (UniqueName: \"kubernetes.io/projected/00725e0e-3a03-4e20-8aa3-50c733c5159d-kube-api-access-z4gvc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:26.436119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.432458 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:50:26.436119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.432581 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 05:50:26.471554 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:50:26.470816 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:50:26.773934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.773901 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" Oct 30 05:50:26.773934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.773920 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn" event={"ID":"00725e0e-3a03-4e20-8aa3-50c733c5159d","Type":"ContainerDied","Data":"e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8"} Oct 30 05:50:26.774202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:26.773998 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e821d7242e8add9a0c2e20cff6a69c55036883fc7f086135b59fe59837892be8" Oct 30 05:50:27.427588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:27.427551 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 05:50:27.427987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:27.427618 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 05:50:28.838543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:28.838515 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:28.839102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:28.839075 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:28.839847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:28.839824 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:50:31.179712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.179677 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf"] Oct 30 05:50:31.180380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180365 6171 cpu_manager.go:414] "containerMap: removing container" podUID="93317926-5246-42e4-b051-0fa3e16d0827" containerName="revoke-keys" Oct 30 05:50:31.180440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180381 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93317926-5246-42e4-b051-0fa3e16d0827" containerName="revoke-keys" Oct 30 05:50:31.180440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180426 6171 cpu_manager.go:414] "containerMap: removing container" podUID="00725e0e-3a03-4e20-8aa3-50c733c5159d" containerName="revoke-keys" Oct 30 05:50:31.180440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180431 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00725e0e-3a03-4e20-8aa3-50c733c5159d" containerName="revoke-keys" Oct 30 05:50:31.180541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180530 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="00725e0e-3a03-4e20-8aa3-50c733c5159d" containerName="revoke-keys" Oct 30 05:50:31.180575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.180549 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="93317926-5246-42e4-b051-0fa3e16d0827" containerName="revoke-keys" Oct 30 05:50:31.185434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.185415 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.203788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.203762 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf"] Oct 30 05:50:31.208647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.208621 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:31.208647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.208645 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:31.208795 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.208713 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:50:31.211678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.211649 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:31.257785 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.257762 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7977\" (UniqueName: \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.257885 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.257812 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.257979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.257961 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.359785 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.359756 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7977\" (UniqueName: \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.359940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.359806 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.359940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.359890 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.360627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.360608 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.362455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.362438 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.401131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.401080 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7977\" (UniqueName: \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\") pod \"maas-api-revoke-sub-6be5590e-kkcjf\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.542453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.542417 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:31.545145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.545098 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm"] Oct 30 05:50:31.551973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.551948 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.554216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.554191 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm"] Oct 30 05:50:31.565399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.565365 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjbsc\" (UniqueName: \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.565507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.565430 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.565573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.565560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.667746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.666782 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.667746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.667069 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjbsc\" (UniqueName: \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.667746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.667105 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.668022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.667952 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.673582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.673536 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.693822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.693801 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjbsc\" (UniqueName: \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\") pod \"maas-api-revoke-sub-133fe481-sktzm\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:31.888880 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:31.888851 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod251c399a_4fb5_4247_b277_c716229a5524.slice/crio-906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d WatchSource:0}: Error finding container 906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d: Status 404 returned error can't find the container with id 906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d Oct 30 05:50:31.891800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.891778 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf"] Oct 30 05:50:31.912660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:31.912638 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:32.220348 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:32.220319 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05263769_0ea2_4786_ad5b_7f3d56bb9f25.slice/crio-277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c WatchSource:0}: Error finding container 277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c: Status 404 returned error can't find the container with id 277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c Oct 30 05:50:32.222838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.222809 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm"] Oct 30 05:50:32.830491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.830401 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" event={"ID":"251c399a-4fb5-4247-b277-c716229a5524","Type":"ContainerStarted","Data":"892e7946ea2137258c733de4cdea250a22c44fb5aabf4b9966ce7045b2a38694"} Oct 30 05:50:32.830491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.830458 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" event={"ID":"251c399a-4fb5-4247-b277-c716229a5524","Type":"ContainerStarted","Data":"906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d"} Oct 30 05:50:32.832187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.832162 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" event={"ID":"05263769-0ea2-4786-ad5b-7f3d56bb9f25","Type":"ContainerStarted","Data":"698378bc9973ef82b9c5b583fad6f8d9db6d25873237a103f8b859710b2c62bc"} Oct 30 05:50:32.832302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.832199 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" event={"ID":"05263769-0ea2-4786-ad5b-7f3d56bb9f25","Type":"ContainerStarted","Data":"277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c"} Oct 30 05:50:32.875323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.875249 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" podStartSLOduration=1.875236568 podStartE2EDuration="1.875236568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:32.870219469 +0000 UTC m=+1091.310596661" watchObservedRunningTime="2026-10-30 05:50:32.875236568 +0000 UTC m=+1091.315613759" Oct 30 05:50:32.913548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:32.913494 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" podStartSLOduration=1.913481513 podStartE2EDuration="1.913481513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:32.909460937 +0000 UTC m=+1091.349838134" watchObservedRunningTime="2026-10-30 05:50:32.913481513 +0000 UTC m=+1091.353858705" Oct 30 05:50:33.844260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:33.844204 6171 generic.go:356] "Generic (PLEG): container finished" podID="251c399a-4fb5-4247-b277-c716229a5524" containerID="892e7946ea2137258c733de4cdea250a22c44fb5aabf4b9966ce7045b2a38694" exitCode=0 Oct 30 05:50:33.844744 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:33.844362 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" event={"ID":"251c399a-4fb5-4247-b277-c716229a5524","Type":"ContainerDied","Data":"892e7946ea2137258c733de4cdea250a22c44fb5aabf4b9966ce7045b2a38694"} Oct 30 05:50:33.846610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:33.846586 6171 generic.go:356] "Generic (PLEG): container finished" podID="05263769-0ea2-4786-ad5b-7f3d56bb9f25" containerID="698378bc9973ef82b9c5b583fad6f8d9db6d25873237a103f8b859710b2c62bc" exitCode=0 Oct 30 05:50:33.846762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:33.846666 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" event={"ID":"05263769-0ea2-4786-ad5b-7f3d56bb9f25","Type":"ContainerDied","Data":"698378bc9973ef82b9c5b583fad6f8d9db6d25873237a103f8b859710b2c62bc"} Oct 30 05:50:34.585862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.585832 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd"] Oct 30 05:50:34.590614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.590599 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.610502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.610474 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd"] Oct 30 05:50:34.767111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.767037 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.767323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.767271 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.767396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.767325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9dkj\" (UniqueName: \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.868560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.868528 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.868982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.868613 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9dkj\" (UniqueName: \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.868982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.868799 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.869551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.869524 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.871117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.871099 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.894196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.894147 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9dkj\" (UniqueName: \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\") pod \"maas-api-revoke-sub-e8c9dbc3-s46qd\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:34.905212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:34.905013 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:35.030803 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.030774 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:35.086074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.086035 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:35.175229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175152 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\" (UniqueName: \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\") pod \"251c399a-4fb5-4247-b277-c716229a5524\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " Oct 30 05:50:35.175383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175247 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\") pod \"251c399a-4fb5-4247-b277-c716229a5524\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " Oct 30 05:50:35.175383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175322 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\") pod \"251c399a-4fb5-4247-b277-c716229a5524\" (UID: \"251c399a-4fb5-4247-b277-c716229a5524\") " Oct 30 05:50:35.175485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175386 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\" (UniqueName: \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\") pod \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " Oct 30 05:50:35.175485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175463 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\") pod \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " Oct 30 05:50:35.175590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.175515 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\") pod \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\" (UID: \"05263769-0ea2-4786-ad5b-7f3d56bb9f25\") " Oct 30 05:50:35.177266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.177238 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca" pod "05263769-0ea2-4786-ad5b-7f3d56bb9f25" (UID: "05263769-0ea2-4786-ad5b-7f3d56bb9f25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:35.178821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.178795 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca" pod "251c399a-4fb5-4247-b277-c716229a5524" (UID: "251c399a-4fb5-4247-b277-c716229a5524"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:35.182345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.182317 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977" pod "251c399a-4fb5-4247-b277-c716229a5524" (UID: "251c399a-4fb5-4247-b277-c716229a5524"). InnerVolumeSpecName "kube-api-access-j7977". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:35.190291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.190266 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca" pod "251c399a-4fb5-4247-b277-c716229a5524" (UID: "251c399a-4fb5-4247-b277-c716229a5524"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:35.191787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.191761 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca" pod "05263769-0ea2-4786-ad5b-7f3d56bb9f25" (UID: "05263769-0ea2-4786-ad5b-7f3d56bb9f25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:35.194553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.194531 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc" pod "05263769-0ea2-4786-ad5b-7f3d56bb9f25" (UID: "05263769-0ea2-4786-ad5b-7f3d56bb9f25"). InnerVolumeSpecName "kube-api-access-rjbsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:35.236428 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:35.236399 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23eeeb8f_54a1_4419_a4af_ac460e6de715.slice/crio-df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3 WatchSource:0}: Error finding container df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3: Status 404 returned error can't find the container with id df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3 Oct 30 05:50:35.238093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.238073 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd"] Oct 30 05:50:35.278582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278557 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/251c399a-4fb5-4247-b277-c716229a5524-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.278582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278582 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/251c399a-4fb5-4247-b277-c716229a5524-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.278722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278592 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjbsc\" (UniqueName: \"kubernetes.io/projected/05263769-0ea2-4786-ad5b-7f3d56bb9f25-kube-api-access-rjbsc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.278722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278600 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.278722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278609 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05263769-0ea2-4786-ad5b-7f3d56bb9f25-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.278722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.278616 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7977\" (UniqueName: \"kubernetes.io/projected/251c399a-4fb5-4247-b277-c716229a5524-kube-api-access-j7977\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.719184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719156 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:50:35.719815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719802 6171 cpu_manager.go:414] "containerMap: removing container" podUID="251c399a-4fb5-4247-b277-c716229a5524" containerName="revoke-keys" Oct 30 05:50:35.719865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719816 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="251c399a-4fb5-4247-b277-c716229a5524" containerName="revoke-keys" Oct 30 05:50:35.719865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719830 6171 cpu_manager.go:414] "containerMap: removing container" podUID="05263769-0ea2-4786-ad5b-7f3d56bb9f25" containerName="revoke-keys" Oct 30 05:50:35.719865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719835 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05263769-0ea2-4786-ad5b-7f3d56bb9f25" containerName="revoke-keys" Oct 30 05:50:35.719968 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719958 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="251c399a-4fb5-4247-b277-c716229a5524" containerName="revoke-keys" Oct 30 05:50:35.720003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.719972 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="05263769-0ea2-4786-ad5b-7f3d56bb9f25" containerName="revoke-keys" Oct 30 05:50:35.724137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.724119 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.736210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.736190 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:50:35.740891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.740842 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.751698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.751673 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:50:35.794439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.794409 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:50:35.864661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.864556 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" Oct 30 05:50:35.864661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.864590 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf" event={"ID":"251c399a-4fb5-4247-b277-c716229a5524","Type":"ContainerDied","Data":"906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d"} Oct 30 05:50:35.864661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.864633 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="906ca12d5a5ee1dd5c748868bf8b1c2a2253e90af56d4ccee194dfe7e0e8c06d" Oct 30 05:50:35.879733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.866115 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" Oct 30 05:50:35.879733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.866150 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm" event={"ID":"05263769-0ea2-4786-ad5b-7f3d56bb9f25","Type":"ContainerDied","Data":"277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c"} Oct 30 05:50:35.879733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.866181 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="277d64b51a4369f5d998c6aafa370476dcde9f7b8b2bb9cf5bfd5770e5cdd53c" Oct 30 05:50:35.879733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.868016 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" event={"ID":"23eeeb8f-54a1-4419-a4af-ac460e6de715","Type":"ContainerStarted","Data":"761c48e4da7a7a500171054ce15e8d1832c8ecdf3d32092f4d64bb7dcc222fce"} Oct 30 05:50:35.879733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.868061 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" event={"ID":"23eeeb8f-54a1-4419-a4af-ac460e6de715","Type":"ContainerStarted","Data":"df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3"} Oct 30 05:50:35.886688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886662 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.886809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886715 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzrqh\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.886809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886785 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.886920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886859 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.886920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886881 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6twq\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.886920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886901 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.886920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886919 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.886939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887091 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887149 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887179 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.887258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887254 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.887410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887278 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.887410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887371 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.887410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887394 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.887610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887417 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.887610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.887436 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.917905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.917855 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" podStartSLOduration=1.917841532 podStartE2EDuration="1.917841532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:35.908430524 +0000 UTC m=+1094.348807718" watchObservedRunningTime="2026-10-30 05:50:35.917841532 +0000 UTC m=+1094.358218727" Oct 30 05:50:35.988561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.988707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988606 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzrqh\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.988707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988632 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.988831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988799 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.988890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988853 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6twq\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.988945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.988945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988894 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988942 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.988980 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989092 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989123 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989193 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989280 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989316 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989354 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989427 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989434 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989458 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989533 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.989896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989559 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.989896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.989793 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.990153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.990077 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.990267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.990242 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.990523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.990501 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.991519 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.991463 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.992716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.992698 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:35.992816 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.992795 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.993017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.993003 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:35.993657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:35.993640 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:36.011756 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.011736 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:36.012094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.012032 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6twq\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:36.012201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.012141 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:36.012354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.012339 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzrqh\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\") pod \"e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:36.044526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.044504 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:36.060481 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.060453 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:36.227798 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.223467 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:50:36.234769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.228559 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.252638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.252541 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:50:36.370509 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:36.370480 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c6f1a87_dc64_4453_9f4f_16e4b28c0436.slice/crio-5f79363b823a39be58e7f8ad49bddca7137d40f7932a6796ab15e8efbc01b6f3 WatchSource:0}: Error finding container 5f79363b823a39be58e7f8ad49bddca7137d40f7932a6796ab15e8efbc01b6f3: Status 404 returned error can't find the container with id 5f79363b823a39be58e7f8ad49bddca7137d40f7932a6796ab15e8efbc01b6f3 Oct 30 05:50:36.372929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.372905 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:50:36.373623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.373592 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.373676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.373656 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.373710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.373683 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.396029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396004 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396136 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396195 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396266 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396307 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396419 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396395 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396481 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh5x2\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396517 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.396698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.396677 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506499 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506553 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh5x2\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506584 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506673 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506818 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506858 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506905 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.506953 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.507005 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.507477 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.507969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.507701 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.515098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.508244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.520797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.518283 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.520797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.518739 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.522034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.521325 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.522034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.521419 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.522034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.521458 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:36.526738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.526521 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.528343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.528295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.546068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.541939 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:50:36.562076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.562026 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.570178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.570148 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh5x2\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\") pod \"e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.865733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.865656 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:36.883606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.883574 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" event={"ID":"a1fd8c76-8881-4965-9942-b5615457dc75","Type":"ContainerStarted","Data":"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd"} Oct 30 05:50:36.884325 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.884299 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" event={"ID":"a1fd8c76-8881-4965-9942-b5615457dc75","Type":"ContainerStarted","Data":"f873409f1cfecc83d02b5cb9408e3bacba8c6c64b3af49552aec15455ea5011a"} Oct 30 05:50:36.889333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.889057 6171 generic.go:356] "Generic (PLEG): container finished" podID="23eeeb8f-54a1-4419-a4af-ac460e6de715" containerID="761c48e4da7a7a500171054ce15e8d1832c8ecdf3d32092f4d64bb7dcc222fce" exitCode=0 Oct 30 05:50:36.889333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.889261 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" event={"ID":"23eeeb8f-54a1-4419-a4af-ac460e6de715","Type":"ContainerDied","Data":"761c48e4da7a7a500171054ce15e8d1832c8ecdf3d32092f4d64bb7dcc222fce"} Oct 30 05:50:36.892904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.892880 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" event={"ID":"6c6f1a87-dc64-4453-9f4f-16e4b28c0436","Type":"ContainerStarted","Data":"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea"} Oct 30 05:50:36.893090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.893070 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" event={"ID":"6c6f1a87-dc64-4453-9f4f-16e4b28c0436","Type":"ContainerStarted","Data":"5f79363b823a39be58e7f8ad49bddca7137d40f7932a6796ab15e8efbc01b6f3"} Oct 30 05:50:36.938724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:36.938674 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podStartSLOduration=1.938656465 podStartE2EDuration="1.938656465s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:36.932476029 +0000 UTC m=+1095.372853222" watchObservedRunningTime="2026-10-30 05:50:36.938656465 +0000 UTC m=+1095.379033652" Oct 30 05:50:37.044922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.044892 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:37.046034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.046007 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:37.046199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.046069 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:37.052432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.052372 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podStartSLOduration=2.052354405 podStartE2EDuration="2.052354405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:37.051146858 +0000 UTC m=+1095.491524050" watchObservedRunningTime="2026-10-30 05:50:37.052354405 +0000 UTC m=+1095.492731596" Oct 30 05:50:37.061279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.061255 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:37.062391 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.062370 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:37.062587 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.062411 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:37.212039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.212010 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:50:37.221748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.219510 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.221748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.219596 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.221748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.219647 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:50:37.451132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.451100 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:50:37.942546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.942507 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" event={"ID":"e7453981-f89e-40b2-8190-105df4cd5a4b","Type":"ContainerStarted","Data":"fd25a347d056784cfaa55fb9a881667649b07719225fd77c9d32247a86b22906"} Oct 30 05:50:37.943394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:37.943099 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" event={"ID":"e7453981-f89e-40b2-8190-105df4cd5a4b","Type":"ContainerStarted","Data":"db303689fc3776f9a10d23f8a542da6bc96ae05fddccde49bbb23426073e793f"} Oct 30 05:50:38.033390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.018692 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podStartSLOduration=2.018671449 podStartE2EDuration="2.018671449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:38.001395561 +0000 UTC m=+1096.441772754" watchObservedRunningTime="2026-10-30 05:50:38.018671449 +0000 UTC m=+1096.459048641" Oct 30 05:50:38.111240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.108206 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.111240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.108268 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:38.140069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.122553 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.140069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.122614 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:38.488332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.488285 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:38.567503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.566556 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\" (UniqueName: \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\") pod \"23eeeb8f-54a1-4419-a4af-ac460e6de715\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " Oct 30 05:50:38.567503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.566594 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\") pod \"23eeeb8f-54a1-4419-a4af-ac460e6de715\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " Oct 30 05:50:38.567503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.566848 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\") pod \"23eeeb8f-54a1-4419-a4af-ac460e6de715\" (UID: \"23eeeb8f-54a1-4419-a4af-ac460e6de715\") " Oct 30 05:50:38.568177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.567819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca" pod "23eeeb8f-54a1-4419-a4af-ac460e6de715" (UID: "23eeeb8f-54a1-4419-a4af-ac460e6de715"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:38.571249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.571209 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca" pod "23eeeb8f-54a1-4419-a4af-ac460e6de715" (UID: "23eeeb8f-54a1-4419-a4af-ac460e6de715"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:38.572807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.572774 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj" pod "23eeeb8f-54a1-4419-a4af-ac460e6de715" (UID: "23eeeb8f-54a1-4419-a4af-ac460e6de715"). InnerVolumeSpecName "kube-api-access-j9dkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:38.670129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.669804 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9dkj\" (UniqueName: \"kubernetes.io/projected/23eeeb8f-54a1-4419-a4af-ac460e6de715-kube-api-access-j9dkj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:38.670129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.669841 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:38.670129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.669857 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/23eeeb8f-54a1-4419-a4af-ac460e6de715-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:38.877781 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.870146 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:38.903071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.901273 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.903071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:38.901389 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:39.083073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.057719 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.083073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.057782 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:39.083073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.062420 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.083073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.062483 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:39.131084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.117609 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" Oct 30 05:50:39.131084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.123515 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd" event={"ID":"23eeeb8f-54a1-4419-a4af-ac460e6de715","Type":"ContainerDied","Data":"df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3"} Oct 30 05:50:39.131084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.123594 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df6259366ad1dd5b5533a19a956c51aa11913b713d3c3684618d839d9b6c4dc3" Oct 30 05:50:39.875692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.875639 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.876003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:39.875965 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:40.035072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.025239 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:50:40.057004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.056950 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.057330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.057274 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:40.066585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.066440 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.066585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.066514 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:40.131130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.130378 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.135343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.135304 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.224279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.224231 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:50:40.340069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.339166 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.415128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.414133 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:40.563070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.562873 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:40.871076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.867455 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.871076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:40.867517 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:41.047071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.046604 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.047071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.046675 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:41.071069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.068927 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.071069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.068983 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:41.135135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.134130 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.135135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.134581 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:41.174072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.163419 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.174072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.163823 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:50:41.327077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.301400 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.327077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.301822 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:50:41.438141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.367864 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.438141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.368581 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:50:41.552285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.552243 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:41.553259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.553235 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:50:41.580257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.580219 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:41.581343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.581316 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:50:41.924072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.909625 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:41.924396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:41.924362 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:42.123469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.123432 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:42.124156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.124121 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:42.152088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.151304 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:42.152088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.151370 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:42.152088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.151865 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.244164 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.222638 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:50:42.320409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.318737 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.372274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.371133 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.601591 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.592913 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:50:42.618014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.617971 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:42.701129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.675898 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:42.953131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.941194 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:42.953131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:42.941258 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:43.080081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.076733 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:43.084168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.084123 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:43.100565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.081164 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:43.129180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.129128 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:43.186955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.186913 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.589624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.234351 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.669447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.246093 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 05:50:43.689164 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.689132 6171 cpu_manager.go:414] "containerMap: removing container" podUID="23eeeb8f-54a1-4419-a4af-ac460e6de715" containerName="revoke-keys" Oct 30 05:50:43.689394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.689377 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23eeeb8f-54a1-4419-a4af-ac460e6de715" containerName="revoke-keys" Oct 30 05:50:43.689747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.689729 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="23eeeb8f-54a1-4419-a4af-ac460e6de715" containerName="revoke-keys" Oct 30 05:50:43.795456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.402309 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:43.829434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.319816 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.593130 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:43.673133 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.021754 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.021810 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.022261 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.044073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.022301 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:44.070034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.069997 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.070442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.070413 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:44.070915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.070891 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.071094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.071072 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:44.163225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.163184 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" podUID="9e53b670-2539-4a61-a1aa-fe5e43005508" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.16:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.342426 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.342314 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 05:50:44.390286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.390252 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.449160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.449125 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.454161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.454123 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.507867 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.507821 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.508223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.508192 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.547443 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.547620 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.547732 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.547827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mzqv\" (UniqueName: \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.548089 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.553070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.548210 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.562078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.554463 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c78066\"/\"distinct-2-w2-c78066-kserve-self-signed-certs\"" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670036 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670134 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mzqv\" (UniqueName: \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670318 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670408 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670454 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.674342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.670538 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.676023 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.676317 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.676653 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.677832 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.677860 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:44.684098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.677902 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.736125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.721287 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.776971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.776903 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:44.790655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.790620 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.843234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.836486 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:44.872869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.872836 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:44.877696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.877665 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:44.885174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.885136 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:44.965698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:44.965617 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mzqv\" (UniqueName: \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\") pod \"distinct-2-w2-c78066-kserve-8449dd6d78-g72b7\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:45.003472 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.003333 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:45.003472 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.003399 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.007665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.006599 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.007665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.006630 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:50:45.007665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.006599 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.074081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.073316 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:45.074081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.073389 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:45.074081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.073672 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:45.074081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.073730 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:45.087082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.087018 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:45.458073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.455358 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.684131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.683136 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.722071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.721338 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.839244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.839185 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:45.866014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.865976 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:45.866313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:45.866269 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:46.009291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.009248 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:50:46.011107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.011072 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:46.014494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.014459 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.045518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.045405 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:46.045518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.045463 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:46.071500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.071463 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:46.071804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.071776 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:46.328274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.328170 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:46.332069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.328511 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.721121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.712178 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:46.883417 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.883377 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:46.883707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.883676 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:46.884268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:46.884237 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.004291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.004149 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:47.004291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.004218 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.034109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.028250 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.034109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.028896 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.034109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.033444 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.073261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.072632 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:47.073261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.072691 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:47.100105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.092718 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:47.100105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.092781 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:47.374312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.323192 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 05:50:47.374312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.323296 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.696339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.691554 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": context deadline exceeded" Oct 30 05:50:47.701238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.701179 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" podUID="9e53b670-2539-4a61-a1aa-fe5e43005508" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.16:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.728082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.725206 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.745237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.745178 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:47.745586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.745463 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:47.915119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.914986 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:50:47.957753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.957713 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:47.958035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:47.958002 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.059010 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.059098 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.059968 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.060005 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.060097 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.062014 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:48.076735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.062784 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:48.125871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.124342 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.125871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.124414 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:48.367263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.332185 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:48.367263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.360656 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:48.975005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.922393 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:50:48.983547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.978385 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:48.983791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.983759 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:49.152487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:48.858870 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:50:49.215082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.178222 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:50:49.215082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.178554 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:50:49.351263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.351223 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:50:49.415125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.412101 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:50:49.425842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.425782 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:49.463742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.463706 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:49.468148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.455379 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t"] Oct 30 05:50:49.645345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.638201 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:49.691184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.689205 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:49.691184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.689279 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:49.955120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.954832 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.000508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.986568 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:50.000508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.987159 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:50:50.000508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.987212 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:50.000508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:49.987276 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:50.157095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.124950 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5mb9\" (UniqueName: \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.195951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.195910 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.338070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.337227 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.338070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.157478 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:50.338070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.337438 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:50.346070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.186116 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:50:50.346070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.338836 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:50:50.346070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.338862 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t"] Oct 30 05:50:50.346070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.338908 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" event={"ID":"7165b02e-9bec-423a-9040-55cae52fe2fb","Type":"ContainerStarted","Data":"686484f29c3ba1c1060cb1a1403eeeca70d992ba631ec291dad18dffcca2eb1b"} Oct 30 05:50:50.346070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.338934 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 05:50:50.361069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.358150 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:50.448564 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.243011 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:50.448835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.448808 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:50.453614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.263944 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:50.453808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.453784 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:50.497998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.497972 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:50:50.498325 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.498264 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 05:50:50.498530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.498520 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:50:50.498663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.498654 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:50:50.519745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.505734 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.531557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.531531 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1354a5\"/\"openshift-service-ca.crt\"" Oct 30 05:50:50.531976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.531947 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1354a5\"/\"default-dockercfg-cnhtk\"" Oct 30 05:50:50.552484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.552457 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-1354a5\"/\"kube-root-ca.crt\"" Oct 30 05:50:50.555841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.555819 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-1354a5\"/\"distinct-2-w6-1354a5-kserve-self-signed-certs\"" Oct 30 05:50:50.663602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.663562 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk"] Oct 30 05:50:50.679316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.679291 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk"] Oct 30 05:50:50.679539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.679525 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:50:50.722066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.680854 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.734619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.734497 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.734619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.734594 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5mb9\" (UniqueName: \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.743117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.692853 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.744566 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.744534 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.746242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.746218 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.767775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.767692 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5mb9\" (UniqueName: \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\") pod \"maas-api-revoke-sub-5f50b80f-fsf5t\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.840336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840305 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj4sz\" (UniqueName: \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.840525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840435 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvct5\" (UniqueName: \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.840525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840472 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.840647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840527 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.840706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840671 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.840771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840755 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.840937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.840921 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.841031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.841020 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.841662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.841645 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.866902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.866848 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:50.867162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.867133 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:50.902269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.902207 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" event={"ID":"7165b02e-9bec-423a-9040-55cae52fe2fb","Type":"ContainerStarted","Data":"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443"} Oct 30 05:50:50.945440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945403 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.945602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945474 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.945684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945598 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.945684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945662 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.945787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945775 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.945840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945800 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.945902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945887 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.945956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.945922 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj4sz\" (UniqueName: \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.946078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.946039 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvct5\" (UniqueName: \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.946899 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.946872 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.947178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.947159 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.948595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.948566 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.948891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.948869 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.951942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.951919 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.952124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.952005 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:50.952322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.952300 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.960605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.960588 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:50.967104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.967032 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podStartSLOduration=8.967020221 podStartE2EDuration="8.967020221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:50.96425729 +0000 UTC m=+1109.404634482" watchObservedRunningTime="2026-10-30 05:50:50.967020221 +0000 UTC m=+1109.407397436" Oct 30 05:50:50.974582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.974562 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj4sz\" (UniqueName: \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\") pod \"maas-api-revoke-sub-b395cac9-d6qpk\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:50.975844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:50.975822 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvct5\" (UniqueName: \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\") pod \"distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:51.037248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.037207 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:51.048277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.047170 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:51.048826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.048482 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:51.048826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.048540 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:51.063013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.062387 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:51.063013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.062739 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body= Oct 30 05:50:51.063013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.062781 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" Oct 30 05:50:51.101120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.099580 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:51.151494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.150207 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:51.469180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.469097 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z"] Oct 30 05:50:51.527545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.527500 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z"] Oct 30 05:50:51.527820 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.527800 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.566560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.566226 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t"] Oct 30 05:50:51.631504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.625680 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.631504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.625846 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlhjf\" (UniqueName: \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.631504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.625933 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.803561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.769720 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlhjf\" (UniqueName: \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.803561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.769854 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.803561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.770125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.803561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.777683 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.826126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.805840 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.868877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.868395 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlhjf\" (UniqueName: \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\") pod \"maas-api-revoke-sub-d0aae7c2-2jh7z\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:51.883223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.883195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:51.883332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.883257 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 05:50:51.884784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.884761 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:51.884945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.884922 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:51.914656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.914625 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" event={"ID":"9f4473e3-3e5c-4e5b-905b-cfe44b517f35","Type":"ContainerStarted","Data":"803db64475fca1edcb0cb2d55a3baada1b7b64ed669c0ea6b236a9aa7023df71"} Oct 30 05:50:51.916239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:51.916214 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" event={"ID":"a59d0e90-3b18-403f-bb1d-ff82b228d3e5","Type":"ContainerStarted","Data":"a552e6190190c1d4ff349e560689f46f8fafe3abf46fc708815797f9575e8276"} Oct 30 05:50:52.013300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.013269 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk"] Oct 30 05:50:52.045786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.045747 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" start-of-body= Oct 30 05:50:52.046278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.045808 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.121:15021/healthz/ready\": dial tcp 10.133.0.121:15021: connect: connection refused" Oct 30 05:50:52.156346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.156136 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:52.576788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.576684 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z"] Oct 30 05:50:52.865635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.865608 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:52.866289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.866261 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body= Oct 30 05:50:52.866866 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.866838 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" Oct 30 05:50:52.868182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.868159 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:50:52.931431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.931343 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" event={"ID":"74a52252-82ac-4256-919d-9c065c3aab64","Type":"ContainerStarted","Data":"047b619cc31bc8eeab4adb7a1b0036d83b891dc9e9ebb06ddc192682b7f3fa51"} Oct 30 05:50:52.933746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.933713 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" event={"ID":"10ed4255-f11d-4ac4-ab20-d476f532db79","Type":"ContainerStarted","Data":"0a73ceb46af4cb7cf8f220dee7e7150439c2532ff2e580940b2123c836ba9c30"} Oct 30 05:50:52.933895 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.933769 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" event={"ID":"10ed4255-f11d-4ac4-ab20-d476f532db79","Type":"ContainerStarted","Data":"61b514256a5f267bb80c0b495985e28f5bc9bc7ea03478cd007981bbfb1e2913"} Oct 30 05:50:52.936829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.936802 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" event={"ID":"9f4473e3-3e5c-4e5b-905b-cfe44b517f35","Type":"ContainerStarted","Data":"ea2376fae66628648ff1afad28a2d77b9945972a2cede9b60f4f7614d4e9c19f"} Oct 30 05:50:52.939672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:52.939643 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" event={"ID":"a59d0e90-3b18-403f-bb1d-ff82b228d3e5","Type":"ContainerStarted","Data":"c14c8f4098cbbf3c5b5510b68195943ee9752dd8cb1e698888e5f5355c904bee"} Oct 30 05:50:53.030350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.023137 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podStartSLOduration=4.023119671 podStartE2EDuration="4.023119671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:53.019603591 +0000 UTC m=+1111.459980780" watchObservedRunningTime="2026-10-30 05:50:53.023119671 +0000 UTC m=+1111.463496863" Oct 30 05:50:53.038660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.038631 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:50:53.074513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.074450 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" podStartSLOduration=3.074409805 podStartE2EDuration="3.074409805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:53.067752688 +0000 UTC m=+1111.508129882" watchObservedRunningTime="2026-10-30 05:50:53.074409805 +0000 UTC m=+1111.514786998" Oct 30 05:50:53.080172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.077446 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:53.098187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.098099 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:50:53.238081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.233214 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" podStartSLOduration=5.233189259 podStartE2EDuration="5.233189259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:53.133190098 +0000 UTC m=+1111.573567291" watchObservedRunningTime="2026-10-30 05:50:53.233189259 +0000 UTC m=+1111.673566490" Oct 30 05:50:53.950393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.950363 6171 generic.go:356] "Generic (PLEG): container finished" podID="a59d0e90-3b18-403f-bb1d-ff82b228d3e5" containerID="c14c8f4098cbbf3c5b5510b68195943ee9752dd8cb1e698888e5f5355c904bee" exitCode=0 Oct 30 05:50:53.950590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.950453 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" event={"ID":"a59d0e90-3b18-403f-bb1d-ff82b228d3e5","Type":"ContainerDied","Data":"c14c8f4098cbbf3c5b5510b68195943ee9752dd8cb1e698888e5f5355c904bee"} Oct 30 05:50:53.952223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.952197 6171 generic.go:356] "Generic (PLEG): container finished" podID="74a52252-82ac-4256-919d-9c065c3aab64" containerID="5c237f25166c77c8ecee58b644110c95ad9bb2ae37b557f7691f478a3d5355c8" exitCode=0 Oct 30 05:50:53.952334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.952286 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" event={"ID":"74a52252-82ac-4256-919d-9c065c3aab64","Type":"ContainerDied","Data":"5c237f25166c77c8ecee58b644110c95ad9bb2ae37b557f7691f478a3d5355c8"} Oct 30 05:50:53.953996 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.953976 6171 generic.go:356] "Generic (PLEG): container finished" podID="10ed4255-f11d-4ac4-ab20-d476f532db79" containerID="0a73ceb46af4cb7cf8f220dee7e7150439c2532ff2e580940b2123c836ba9c30" exitCode=0 Oct 30 05:50:53.954115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:53.954061 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" event={"ID":"10ed4255-f11d-4ac4-ab20-d476f532db79","Type":"ContainerDied","Data":"0a73ceb46af4cb7cf8f220dee7e7150439c2532ff2e580940b2123c836ba9c30"} Oct 30 05:50:54.907209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:54.907159 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:50:54.907706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:54.907213 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:55.107280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.104194 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:55.132770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.132699 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:55.190197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.189981 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:55.195184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.195006 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:50:55.299963 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.299930 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:55.371753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.371729 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:55.378275 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.378249 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:55.426101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426027 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\" (UniqueName: \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\") pod \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " Oct 30 05:50:55.426280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426114 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\") pod \"10ed4255-f11d-4ac4-ab20-d476f532db79\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " Oct 30 05:50:55.426517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426487 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca" pod "10ed4255-f11d-4ac4-ab20-d476f532db79" (UID: "10ed4255-f11d-4ac4-ab20-d476f532db79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:55.426618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426521 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\") pod \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " Oct 30 05:50:55.426676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426618 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\" (UniqueName: \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\") pod \"74a52252-82ac-4256-919d-9c065c3aab64\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " Oct 30 05:50:55.426676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\") pod \"74a52252-82ac-4256-919d-9c065c3aab64\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " Oct 30 05:50:55.426861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426844 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\") pod \"10ed4255-f11d-4ac4-ab20-d476f532db79\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " Oct 30 05:50:55.426977 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.426941 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\" (UniqueName: \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\") pod \"10ed4255-f11d-4ac4-ab20-d476f532db79\" (UID: \"10ed4255-f11d-4ac4-ab20-d476f532db79\") " Oct 30 05:50:55.427063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.427024 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\") pod \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\" (UID: \"a59d0e90-3b18-403f-bb1d-ff82b228d3e5\") " Oct 30 05:50:55.427127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.427080 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\") pod \"74a52252-82ac-4256-919d-9c065c3aab64\" (UID: \"74a52252-82ac-4256-919d-9c065c3aab64\") " Oct 30 05:50:55.427703 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.427682 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.428974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.428542 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca" pod "74a52252-82ac-4256-919d-9c065c3aab64" (UID: "74a52252-82ac-4256-919d-9c065c3aab64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:55.429310 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.429273 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca" pod "a59d0e90-3b18-403f-bb1d-ff82b228d3e5" (UID: "a59d0e90-3b18-403f-bb1d-ff82b228d3e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:55.429807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.429773 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9" pod "a59d0e90-3b18-403f-bb1d-ff82b228d3e5" (UID: "a59d0e90-3b18-403f-bb1d-ff82b228d3e5"). InnerVolumeSpecName "kube-api-access-d5mb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:55.430005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.429985 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca" pod "74a52252-82ac-4256-919d-9c065c3aab64" (UID: "74a52252-82ac-4256-919d-9c065c3aab64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:55.430834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.430799 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf" pod "74a52252-82ac-4256-919d-9c065c3aab64" (UID: "74a52252-82ac-4256-919d-9c065c3aab64"). InnerVolumeSpecName "kube-api-access-qlhjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:55.430950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.430928 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca" pod "a59d0e90-3b18-403f-bb1d-ff82b228d3e5" (UID: "a59d0e90-3b18-403f-bb1d-ff82b228d3e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:55.431659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.431635 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca" pod "10ed4255-f11d-4ac4-ab20-d476f532db79" (UID: "10ed4255-f11d-4ac4-ab20-d476f532db79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:55.432473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.432452 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz" pod "10ed4255-f11d-4ac4-ab20-d476f532db79" (UID: "10ed4255-f11d-4ac4-ab20-d476f532db79"). InnerVolumeSpecName "kube-api-access-kj4sz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:55.529770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529741 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10ed4255-f11d-4ac4-ab20-d476f532db79-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529768 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj4sz\" (UniqueName: \"kubernetes.io/projected/10ed4255-f11d-4ac4-ab20-d476f532db79-kube-api-access-kj4sz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529778 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529787 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529796 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5mb9\" (UniqueName: \"kubernetes.io/projected/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-kube-api-access-d5mb9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529804 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59d0e90-3b18-403f-bb1d-ff82b228d3e5-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529812 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlhjf\" (UniqueName: \"kubernetes.io/projected/74a52252-82ac-4256-919d-9c065c3aab64-kube-api-access-qlhjf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.529950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.529821 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74a52252-82ac-4256-919d-9c065c3aab64-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:50:55.996945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.996919 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" Oct 30 05:50:55.997438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.996961 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk" event={"ID":"10ed4255-f11d-4ac4-ab20-d476f532db79","Type":"ContainerDied","Data":"61b514256a5f267bb80c0b495985e28f5bc9bc7ea03478cd007981bbfb1e2913"} Oct 30 05:50:55.997438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.997013 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61b514256a5f267bb80c0b495985e28f5bc9bc7ea03478cd007981bbfb1e2913" Oct 30 05:50:55.998593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.998571 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" Oct 30 05:50:55.998739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.998597 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t" event={"ID":"a59d0e90-3b18-403f-bb1d-ff82b228d3e5","Type":"ContainerDied","Data":"a552e6190190c1d4ff349e560689f46f8fafe3abf46fc708815797f9575e8276"} Oct 30 05:50:55.998739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:55.998643 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a552e6190190c1d4ff349e560689f46f8fafe3abf46fc708815797f9575e8276" Oct 30 05:50:56.001217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:56.001198 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" Oct 30 05:50:56.001339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:56.001219 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z" event={"ID":"74a52252-82ac-4256-919d-9c065c3aab64","Type":"ContainerDied","Data":"047b619cc31bc8eeab4adb7a1b0036d83b891dc9e9ebb06ddc192682b7f3fa51"} Oct 30 05:50:56.001339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:56.001258 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="047b619cc31bc8eeab4adb7a1b0036d83b891dc9e9ebb06ddc192682b7f3fa51" Oct 30 05:50:56.012950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:56.012926 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:50:56.382973 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:50:56.382894 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:50:57.881344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.881311 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v"] Oct 30 05:50:57.882008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.881990 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a59d0e90-3b18-403f-bb1d-ff82b228d3e5" containerName="revoke-keys" Oct 30 05:50:57.882008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882007 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59d0e90-3b18-403f-bb1d-ff82b228d3e5" containerName="revoke-keys" Oct 30 05:50:57.882218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882023 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74a52252-82ac-4256-919d-9c065c3aab64" containerName="revoke-keys" Oct 30 05:50:57.882218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882028 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74a52252-82ac-4256-919d-9c065c3aab64" containerName="revoke-keys" Oct 30 05:50:57.882218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882111 6171 cpu_manager.go:414] "containerMap: removing container" podUID="10ed4255-f11d-4ac4-ab20-d476f532db79" containerName="revoke-keys" Oct 30 05:50:57.882218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882121 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10ed4255-f11d-4ac4-ab20-d476f532db79" containerName="revoke-keys" Oct 30 05:50:57.882439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882246 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="10ed4255-f11d-4ac4-ab20-d476f532db79" containerName="revoke-keys" Oct 30 05:50:57.882439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882257 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59d0e90-3b18-403f-bb1d-ff82b228d3e5" containerName="revoke-keys" Oct 30 05:50:57.882439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.882267 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="74a52252-82ac-4256-919d-9c065c3aab64" containerName="revoke-keys" Oct 30 05:50:57.893508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.893485 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:57.907094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.907064 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v"] Oct 30 05:50:57.910818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.910792 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:57.911276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.911107 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:57.911407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.911190 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:50:57.919909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.919892 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:57.970327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.970298 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:57.970498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.970353 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjh4\" (UniqueName: \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:57.970498 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:57.970458 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.078077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.075183 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.078077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.075521 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.078077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.075591 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjh4\" (UniqueName: \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.078077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.076925 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.101822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.093609 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.120107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.119946 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjh4\" (UniqueName: \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\") pod \"maas-api-revoke-sub-466e6410-lgf5v\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.216890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.216808 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:50:58.523740 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:50:58.523710 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06de76a3_f362_4a1a_aae2_663e55a185de.slice/crio-6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0 WatchSource:0}: Error finding container 6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0: Status 404 returned error can't find the container with id 6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0 Oct 30 05:50:58.526432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:58.526409 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v"] Oct 30 05:50:59.029998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:59.029963 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" event={"ID":"06de76a3-f362-4a1a-aae2-663e55a185de","Type":"ContainerStarted","Data":"554733eb8a0415f4277cc4eada073c74cf75fe5dfd68ebb9833a22b32b910693"} Oct 30 05:50:59.030427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:59.030060 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" event={"ID":"06de76a3-f362-4a1a-aae2-663e55a185de","Type":"ContainerStarted","Data":"6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0"} Oct 30 05:50:59.070913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:50:59.070827 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" podStartSLOduration=2.070814515 podStartE2EDuration="2.070814515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:59.069167937 +0000 UTC m=+1117.509545128" watchObservedRunningTime="2026-10-30 05:50:59.070814515 +0000 UTC m=+1117.511191707" Oct 30 05:51:00.041471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.041439 6171 generic.go:356] "Generic (PLEG): container finished" podID="06de76a3-f362-4a1a-aae2-663e55a185de" containerID="554733eb8a0415f4277cc4eada073c74cf75fe5dfd68ebb9833a22b32b910693" exitCode=0 Oct 30 05:51:00.041888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.041529 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" event={"ID":"06de76a3-f362-4a1a-aae2-663e55a185de","Type":"ContainerDied","Data":"554733eb8a0415f4277cc4eada073c74cf75fe5dfd68ebb9833a22b32b910693"} Oct 30 05:51:00.729131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.729103 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc"] Oct 30 05:51:00.733961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.733945 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.753524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.753500 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc"] Oct 30 05:51:00.810388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.810355 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.810528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.810491 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4k56\" (UniqueName: \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.810811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.810793 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.912570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.912547 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.912708 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.912598 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.912708 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.912639 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4k56\" (UniqueName: \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.913281 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.913250 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.915070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.915021 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:00.936087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:00.936038 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4k56\" (UniqueName: \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\") pod \"maas-api-revoke-sub-58c83b18-dn8qc\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:01.038792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.038725 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:01.048172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.048145 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:01.052832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.052814 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:01.070632 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.070606 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:01.412750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.412722 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:51:01.492623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.492586 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\") pod \"06de76a3-f362-4a1a-aae2-663e55a185de\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " Oct 30 05:51:01.492789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.492727 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\" (UniqueName: \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\") pod \"06de76a3-f362-4a1a-aae2-663e55a185de\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " Oct 30 05:51:01.492789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.492773 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\") pod \"06de76a3-f362-4a1a-aae2-663e55a185de\" (UID: \"06de76a3-f362-4a1a-aae2-663e55a185de\") " Oct 30 05:51:01.494024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.493569 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca" pod "06de76a3-f362-4a1a-aae2-663e55a185de" (UID: "06de76a3-f362-4a1a-aae2-663e55a185de"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:01.494024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.493971 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:01.495169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.495137 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca" pod "06de76a3-f362-4a1a-aae2-663e55a185de" (UID: "06de76a3-f362-4a1a-aae2-663e55a185de"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:01.495618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.495597 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4" pod "06de76a3-f362-4a1a-aae2-663e55a185de" (UID: "06de76a3-f362-4a1a-aae2-663e55a185de"). InnerVolumeSpecName "kube-api-access-xcjh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:01.595671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.595593 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06de76a3-f362-4a1a-aae2-663e55a185de-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:01.595671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.595625 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjh4\" (UniqueName: \"kubernetes.io/projected/06de76a3-f362-4a1a-aae2-663e55a185de-kube-api-access-xcjh4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:01.600568 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:01.600541 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod396ea7be_2d9c_480e_b3eb_fecb7e47e379.slice/crio-a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d WatchSource:0}: Error finding container a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d: Status 404 returned error can't find the container with id a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d Oct 30 05:51:01.602879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.602858 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc"] Oct 30 05:51:01.775598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.775564 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m"] Oct 30 05:51:01.776329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.776315 6171 cpu_manager.go:414] "containerMap: removing container" podUID="06de76a3-f362-4a1a-aae2-663e55a185de" containerName="revoke-keys" Oct 30 05:51:01.776763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.776331 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06de76a3-f362-4a1a-aae2-663e55a185de" containerName="revoke-keys" Oct 30 05:51:01.776763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.776489 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="06de76a3-f362-4a1a-aae2-663e55a185de" containerName="revoke-keys" Oct 30 05:51:01.781827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.781808 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:01.803029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.803004 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m"] Oct 30 05:51:01.901232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.901206 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:01.901424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.901402 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95x5t\" (UniqueName: \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:01.901686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:01.901655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.003089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.003016 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.003284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.003147 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.003354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.003297 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95x5t\" (UniqueName: \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.003824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.003797 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.005428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.005410 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.026663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.026636 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95x5t\" (UniqueName: \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\") pod \"maas-api-revoke-sub-461da871-67j5m\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.066892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.066861 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" Oct 30 05:51:02.066892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.066882 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v" event={"ID":"06de76a3-f362-4a1a-aae2-663e55a185de","Type":"ContainerDied","Data":"6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0"} Oct 30 05:51:02.067390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.066936 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fcaa4b2f89eaa9c5e1814ba0938e4e9e87dcc7eb6b6abbadb4dbae617d036e0" Oct 30 05:51:02.070107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.068853 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" event={"ID":"396ea7be-2d9c-480e-b3eb-fecb7e47e379","Type":"ContainerStarted","Data":"23fa15c954f414e33eb0dc8000df884b275d60506770edb0c41329862f850276"} Oct 30 05:51:02.070107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.068902 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" event={"ID":"396ea7be-2d9c-480e-b3eb-fecb7e47e379","Type":"ContainerStarted","Data":"a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d"} Oct 30 05:51:02.111990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.111960 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:02.115726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.115604 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" podStartSLOduration=2.115590316 podStartE2EDuration="2.115590316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:02.108690194 +0000 UTC m=+1120.549067385" watchObservedRunningTime="2026-10-30 05:51:02.115590316 +0000 UTC m=+1120.555967507" Oct 30 05:51:02.420771 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:02.420739 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a8a75e6_7355_48fa_8a01_d0d0409e638c.slice/crio-9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb WatchSource:0}: Error finding container 9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb: Status 404 returned error can't find the container with id 9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb Oct 30 05:51:02.423640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:02.423623 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m"] Oct 30 05:51:03.080380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:03.080338 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" event={"ID":"9a8a75e6-7355-48fa-8a01-d0d0409e638c","Type":"ContainerStarted","Data":"5c20a0533a8c353b2ba9977f729b88f86a9482bf50124321ccbd5f558869a164"} Oct 30 05:51:03.080853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:03.080412 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" event={"ID":"9a8a75e6-7355-48fa-8a01-d0d0409e638c","Type":"ContainerStarted","Data":"9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb"} Oct 30 05:51:03.082268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:03.082236 6171 generic.go:356] "Generic (PLEG): container finished" podID="396ea7be-2d9c-480e-b3eb-fecb7e47e379" containerID="23fa15c954f414e33eb0dc8000df884b275d60506770edb0c41329862f850276" exitCode=0 Oct 30 05:51:03.082404 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:03.082313 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" event={"ID":"396ea7be-2d9c-480e-b3eb-fecb7e47e379","Type":"ContainerDied","Data":"23fa15c954f414e33eb0dc8000df884b275d60506770edb0c41329862f850276"} Oct 30 05:51:03.146544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:03.146449 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" podStartSLOduration=2.146434662 podStartE2EDuration="2.146434662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:03.140983237 +0000 UTC m=+1121.581360430" watchObservedRunningTime="2026-10-30 05:51:03.146434662 +0000 UTC m=+1121.586811851" Oct 30 05:51:04.093743 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.093709 6171 generic.go:356] "Generic (PLEG): container finished" podID="9a8a75e6-7355-48fa-8a01-d0d0409e638c" containerID="5c20a0533a8c353b2ba9977f729b88f86a9482bf50124321ccbd5f558869a164" exitCode=0 Oct 30 05:51:04.094201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.093787 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" event={"ID":"9a8a75e6-7355-48fa-8a01-d0d0409e638c","Type":"ContainerDied","Data":"5c20a0533a8c353b2ba9977f729b88f86a9482bf50124321ccbd5f558869a164"} Oct 30 05:51:04.276423 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.276395 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:04.354145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.354036 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\") pod \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " Oct 30 05:51:04.354275 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.354225 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\") pod \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " Oct 30 05:51:04.354444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.354427 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\" (UniqueName: \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\") pod \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\" (UID: \"396ea7be-2d9c-480e-b3eb-fecb7e47e379\") " Oct 30 05:51:04.354624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.354597 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca" pod "396ea7be-2d9c-480e-b3eb-fecb7e47e379" (UID: "396ea7be-2d9c-480e-b3eb-fecb7e47e379"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:04.355182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.355157 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:04.356330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.356306 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca" pod "396ea7be-2d9c-480e-b3eb-fecb7e47e379" (UID: "396ea7be-2d9c-480e-b3eb-fecb7e47e379"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:04.356641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.356625 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56" pod "396ea7be-2d9c-480e-b3eb-fecb7e47e379" (UID: "396ea7be-2d9c-480e-b3eb-fecb7e47e379"). InnerVolumeSpecName "kube-api-access-v4k56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:04.458705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.458673 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4k56\" (UniqueName: \"kubernetes.io/projected/396ea7be-2d9c-480e-b3eb-fecb7e47e379-kube-api-access-v4k56\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:04.458958 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:04.458933 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/396ea7be-2d9c-480e-b3eb-fecb7e47e379-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:05.150069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.149445 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" Oct 30 05:51:05.150976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.150107 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc" event={"ID":"396ea7be-2d9c-480e-b3eb-fecb7e47e379","Type":"ContainerDied","Data":"a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d"} Oct 30 05:51:05.150976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.150166 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5da8b6384b4a87b2b41b1cd4daf668f92464d06f495ac17abdad446526c522d" Oct 30 05:51:05.492560 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.492532 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:05.612107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.611545 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\") pod \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " Oct 30 05:51:05.627073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.621372 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\") pod \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " Oct 30 05:51:05.627073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.621517 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\" (UniqueName: \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\") pod \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\" (UID: \"9a8a75e6-7355-48fa-8a01-d0d0409e638c\") " Oct 30 05:51:05.656130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.653816 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca" pod "9a8a75e6-7355-48fa-8a01-d0d0409e638c" (UID: "9a8a75e6-7355-48fa-8a01-d0d0409e638c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:05.656130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.654604 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:06.006079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:05.986710 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca" pod "9a8a75e6-7355-48fa-8a01-d0d0409e638c" (UID: "9a8a75e6-7355-48fa-8a01-d0d0409e638c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:06.037256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:06.033702 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t" pod "9a8a75e6-7355-48fa-8a01-d0d0409e638c" (UID: "9a8a75e6-7355-48fa-8a01-d0d0409e638c"). InnerVolumeSpecName "kube-api-access-95x5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:06.065086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:06.053563 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95x5t\" (UniqueName: \"kubernetes.io/projected/9a8a75e6-7355-48fa-8a01-d0d0409e638c-kube-api-access-95x5t\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:06.065086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:06.053599 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9a8a75e6-7355-48fa-8a01-d0d0409e638c-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:07.058067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.049267 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:51:07.096627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.059703 6171 cpu_manager.go:414] "containerMap: removing container" podUID="396ea7be-2d9c-480e-b3eb-fecb7e47e379" containerName="revoke-keys" Oct 30 05:51:07.097682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.097657 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="396ea7be-2d9c-480e-b3eb-fecb7e47e379" containerName="revoke-keys" Oct 30 05:51:07.097926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.097912 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9a8a75e6-7355-48fa-8a01-d0d0409e638c" containerName="revoke-keys" Oct 30 05:51:07.098017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.098007 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a8a75e6-7355-48fa-8a01-d0d0409e638c" containerName="revoke-keys" Oct 30 05:51:07.098425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.098403 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a8a75e6-7355-48fa-8a01-d0d0409e638c" containerName="revoke-keys" Oct 30 05:51:07.098574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.098561 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="396ea7be-2d9c-480e-b3eb-fecb7e47e379" containerName="revoke-keys" Oct 30 05:51:07.125113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.073458 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.162068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.049338 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.164289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.143726 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" Oct 30 05:51:07.293096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.292985 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m" event={"ID":"9a8a75e6-7355-48fa-8a01-d0d0409e638c","Type":"ContainerDied","Data":"9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb"} Oct 30 05:51:07.293321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.293300 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9bd5f9df7d077c605dca237c8e0476653cc8e7199c1f47f6bd3824344b5824eb" Oct 30 05:51:07.293440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.293426 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:51:07.293667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.293649 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.410219 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.410171 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474475 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474594 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474662 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9bz5\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474730 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474931 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474964 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.474993 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.477180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.475033 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.578366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.578264 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.579496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.579474 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.601185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.601148 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.601444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.601424 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.601647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.601630 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.601763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.601747 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.601929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.601911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.602456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.579412 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.602750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.602726 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9bz5\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.602946 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.602927 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.603555 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.603534 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.604305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.604285 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.619270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.619235 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": context deadline exceeded" Oct 30 05:51:07.624067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.620135 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.643069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.624675 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.643294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.628375 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.643464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.631691 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.646492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.646182 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.646492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.646324 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.667072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.660111 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.697105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.697071 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9bz5\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\") pod \"e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:07.753067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.745423 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.758917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.758121 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:07.964124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:07.963336 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:08.086078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.079306 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.416094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.405847 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.416094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.406618 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:08.416094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.407623 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:08.416094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.407736 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:51:08.416599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.416565 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.433484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.433452 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:51:08.675475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.668453 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:08.675475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.668729 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.675475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.669256 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:51:08.675475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.669310 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:51:08.767518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.748498 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.767518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.748886 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:08.767518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.762463 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:08.767518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:08.763016 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:51:09.207988 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.149195 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.257204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.257176 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:09.462454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.412597 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.462454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.412892 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:09.466831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.464022 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:51:09.481831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.472140 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.707103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.679444 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:09.707103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.683127 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:09.707103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.683176 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.746527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.678596 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.753146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.753092 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:09.786282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.786224 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:09.867099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.863916 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:09.867099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.864009 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:09.867099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:09.864075 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:10.244206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.207507 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:51:10.259188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.259128 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:10.418751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.418709 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:10.558034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.442365 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:10.567373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.482118 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:10.682069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.681194 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:10.757086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.753140 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:51:10.772085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.770334 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:10.853365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.853315 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:10.937123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:10.936640 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" event={"ID":"651a1fce-06de-4467-8bcd-170ebc160cd5","Type":"ContainerStarted","Data":"be252b5e8f983ba4c4d9d1e45f21c28fae2fea5fe05c6e280c6c2ef0773bdc9b"} Oct 30 05:51:11.395707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.384506 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.561070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.553198 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.589361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.558571 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.589361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.587087 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.845084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.770843 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:11.909079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:11.908152 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.166268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.071840 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:51:12.180142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.091615 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.282086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.229304 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": context deadline exceeded" Oct 30 05:51:12.538070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.537417 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:12.589249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.589200 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.614186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.591124 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:12.614531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.591194 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.721222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.721174 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" podUID="9e53b670-2539-4a61-a1aa-fe5e43005508" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.16:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:12.855843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.855627 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:13.046333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.891456 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" podUID="3e622128-5588-4764-91dd-f7759834b7eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": context deadline exceeded" Oct 30 05:51:13.046333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.927301 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t" podUID="653bfbea-50ba-4751-b13e-da9ef57544bd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:13.046333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:12.939671 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:13.099773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:13.099152 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:13.420149 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:51:13.418523 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.549s" Oct 30 05:51:13.420149 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:13.418598 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:51:13.529371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:13.520822 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:51:13.530901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:13.530867 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:51:14.085822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.085772 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:51:14.153579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.103670 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:14.154396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.103709 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:14.154862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.103725 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:14.155232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.103740 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:14.214091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.184866 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:14.375984 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.375560 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" event={"ID":"651a1fce-06de-4467-8bcd-170ebc160cd5","Type":"ContainerStarted","Data":"184f75a6035be7db3535e1a9508141e7fb640cea8cecbb66a5f43cab8c20cad6"} Oct 30 05:51:14.482816 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.482111 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podStartSLOduration=8.482091414 podStartE2EDuration="8.482091414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:14.324735259 +0000 UTC m=+1132.765112485" watchObservedRunningTime="2026-10-30 05:51:14.482091414 +0000 UTC m=+1132.922468612" Oct 30 05:51:14.569067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.539229 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:14.569067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.560388 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:51:14.575067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.570827 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:14.584704 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.584680 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:51:14.659589 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.659506 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:51:14.664373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.664345 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:14.964891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.964814 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:14.965947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.965922 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:14.966091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:14.965979 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:15.032860 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.032824 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84"] Oct 30 05:51:15.038918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.038897 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.039347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.039318 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84"] Oct 30 05:51:15.054640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.054614 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:15.054823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.054808 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:15.054942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.054930 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:15.055017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.054994 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:51:15.131496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.131463 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcl2r\" (UniqueName: \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.131852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.131587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.131852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.131686 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.233026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.232993 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcl2r\" (UniqueName: \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.233254 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.233149 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.233332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.233275 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.234128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.234104 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.235640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.235610 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.256855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.256830 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcl2r\" (UniqueName: \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\") pod \"maas-api-revoke-sub-8df2ee44-54j84\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.359593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.359556 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:15.478999 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.478967 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5"] Oct 30 05:51:15.488429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.488355 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.524663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.524639 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5"] Oct 30 05:51:15.548215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.548186 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.548388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.548239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jbv\" (UniqueName: \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.548466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.548384 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.649993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.649956 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.650229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.650163 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.650229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.650197 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jbv\" (UniqueName: \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.651033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.651011 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.653333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.653314 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.673991 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.673965 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jbv\" (UniqueName: \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\") pod \"maas-api-revoke-sub-28004554-hf7p5\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.812677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.812646 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84"] Oct 30 05:51:15.813513 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:15.813490 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod952c2fa8_520f_4e58_9265_49463debc9e8.slice/crio-0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53 WatchSource:0}: Error finding container 0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53: Status 404 returned error can't find the container with id 0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53 Oct 30 05:51:15.843310 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.843292 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:15.964705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.964674 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:15.964865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:15.964734 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:16.125819 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:16.125739 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" event={"ID":"952c2fa8-520f-4e58-9265-49463debc9e8","Type":"ContainerStarted","Data":"0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53"} Oct 30 05:51:16.175792 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:16.175760 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefea9f9d_5603_4c46_a2a1_746102dcb82f.slice/crio-cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701 WatchSource:0}: Error finding container cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701: Status 404 returned error can't find the container with id cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701 Oct 30 05:51:16.178752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:16.178730 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5"] Oct 30 05:51:16.977487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:16.971191 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:16.977487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:16.971271 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:17.138247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.138204 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" event={"ID":"efea9f9d-5603-4c46-a2a1-746102dcb82f","Type":"ContainerStarted","Data":"a1e32948077ada63724672c8959c17fb2ee8359635ce22b54744d8de7cb939f3"} Oct 30 05:51:17.138446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.138268 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" event={"ID":"efea9f9d-5603-4c46-a2a1-746102dcb82f","Type":"ContainerStarted","Data":"cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701"} Oct 30 05:51:17.140084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.140007 6171 generic.go:356] "Generic (PLEG): container finished" podID="952c2fa8-520f-4e58-9265-49463debc9e8" containerID="ea50dfffc0af5b3bb858bcea7416576f24a217ebe02921c95a3eb851e451e465" exitCode=0 Oct 30 05:51:17.140194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.140072 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" event={"ID":"952c2fa8-520f-4e58-9265-49463debc9e8","Type":"ContainerDied","Data":"ea50dfffc0af5b3bb858bcea7416576f24a217ebe02921c95a3eb851e451e465"} Oct 30 05:51:17.278718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.278670 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" podStartSLOduration=2.278655679 podStartE2EDuration="2.278655679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:17.207650944 +0000 UTC m=+1135.648028137" watchObservedRunningTime="2026-10-30 05:51:17.278655679 +0000 UTC m=+1135.719032871" Oct 30 05:51:17.872516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.872484 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2"] Oct 30 05:51:17.877451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.877427 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:17.918531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.918482 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2"] Oct 30 05:51:17.964701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.964670 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:17.964831 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:17.964774 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:18.021403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.021370 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.021561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.021417 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.021561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.021509 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v24v\" (UniqueName: \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.123707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.123628 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.123707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.123674 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.123880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.123734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v24v\" (UniqueName: \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.124500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.124478 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.126142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.126121 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.147230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.147202 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v24v\" (UniqueName: \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\") pod \"maas-api-revoke-sub-3c8a7907-j5pm2\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.151506 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.151478 6171 generic.go:356] "Generic (PLEG): container finished" podID="efea9f9d-5603-4c46-a2a1-746102dcb82f" containerID="a1e32948077ada63724672c8959c17fb2ee8359635ce22b54744d8de7cb939f3" exitCode=0 Oct 30 05:51:18.151601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.151568 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" event={"ID":"efea9f9d-5603-4c46-a2a1-746102dcb82f","Type":"ContainerDied","Data":"a1e32948077ada63724672c8959c17fb2ee8359635ce22b54744d8de7cb939f3"} Oct 30 05:51:18.227402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.227372 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:18.376668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.376612 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:18.534420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.534392 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\") pod \"952c2fa8-520f-4e58-9265-49463debc9e8\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " Oct 30 05:51:18.534582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.534439 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\" (UniqueName: \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\") pod \"952c2fa8-520f-4e58-9265-49463debc9e8\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " Oct 30 05:51:18.534641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.534625 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\") pod \"952c2fa8-520f-4e58-9265-49463debc9e8\" (UID: \"952c2fa8-520f-4e58-9265-49463debc9e8\") " Oct 30 05:51:18.534998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.534964 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca" pod "952c2fa8-520f-4e58-9265-49463debc9e8" (UID: "952c2fa8-520f-4e58-9265-49463debc9e8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:18.535585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.535558 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:18.536653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.536631 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca" pod "952c2fa8-520f-4e58-9265-49463debc9e8" (UID: "952c2fa8-520f-4e58-9265-49463debc9e8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:18.537420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.537399 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r" pod "952c2fa8-520f-4e58-9265-49463debc9e8" (UID: "952c2fa8-520f-4e58-9265-49463debc9e8"). InnerVolumeSpecName "kube-api-access-zcl2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:18.656122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.655751 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2"] Oct 30 05:51:18.692084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.686428 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/952c2fa8-520f-4e58-9265-49463debc9e8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:18.692335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.692313 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcl2r\" (UniqueName: \"kubernetes.io/projected/952c2fa8-520f-4e58-9265-49463debc9e8-kube-api-access-zcl2r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:18.964971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.964900 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:18.964971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:18.964958 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:19.162614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.162580 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" event={"ID":"952c2fa8-520f-4e58-9265-49463debc9e8","Type":"ContainerDied","Data":"0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53"} Oct 30 05:51:19.162614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.162603 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84" Oct 30 05:51:19.162857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.162661 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0da167be0ba0dde835542e767ccca597f67824a438b6e34c01c1f939425c4b53" Oct 30 05:51:19.164190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.164171 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" event={"ID":"f70eb699-96ee-43e2-bc52-d3eb63dfe83a","Type":"ContainerStarted","Data":"65f16666f28440121e11f853bac0d1911034448b78fc3e0ca9698a5391059ba1"} Oct 30 05:51:19.388974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.388953 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:19.518528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.518500 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\" (UniqueName: \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\") pod \"efea9f9d-5603-4c46-a2a1-746102dcb82f\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " Oct 30 05:51:19.518699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.518553 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\") pod \"efea9f9d-5603-4c46-a2a1-746102dcb82f\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " Oct 30 05:51:19.518964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.518942 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\") pod \"efea9f9d-5603-4c46-a2a1-746102dcb82f\" (UID: \"efea9f9d-5603-4c46-a2a1-746102dcb82f\") " Oct 30 05:51:19.519395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.519369 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca" pod "efea9f9d-5603-4c46-a2a1-746102dcb82f" (UID: "efea9f9d-5603-4c46-a2a1-746102dcb82f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:19.519634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.519606 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:19.520784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.520761 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv" pod "efea9f9d-5603-4c46-a2a1-746102dcb82f" (UID: "efea9f9d-5603-4c46-a2a1-746102dcb82f"). InnerVolumeSpecName "kube-api-access-92jbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:19.521174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.521149 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca" pod "efea9f9d-5603-4c46-a2a1-746102dcb82f" (UID: "efea9f9d-5603-4c46-a2a1-746102dcb82f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:19.621712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.621681 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92jbv\" (UniqueName: \"kubernetes.io/projected/efea9f9d-5603-4c46-a2a1-746102dcb82f-kube-api-access-92jbv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:19.621877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.621717 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efea9f9d-5603-4c46-a2a1-746102dcb82f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:19.964550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.964472 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" start-of-body= Oct 30 05:51:19.964550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:19.964530 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.132:15021/healthz/ready\": dial tcp 10.133.0.132:15021: connect: connection refused" Oct 30 05:51:20.086538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.086504 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd"] Oct 30 05:51:20.087232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087218 6171 cpu_manager.go:414] "containerMap: removing container" podUID="952c2fa8-520f-4e58-9265-49463debc9e8" containerName="revoke-keys" Oct 30 05:51:20.087301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087233 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="952c2fa8-520f-4e58-9265-49463debc9e8" containerName="revoke-keys" Oct 30 05:51:20.087301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087247 6171 cpu_manager.go:414] "containerMap: removing container" podUID="efea9f9d-5603-4c46-a2a1-746102dcb82f" containerName="revoke-keys" Oct 30 05:51:20.087301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087251 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efea9f9d-5603-4c46-a2a1-746102dcb82f" containerName="revoke-keys" Oct 30 05:51:20.087394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087369 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="952c2fa8-520f-4e58-9265-49463debc9e8" containerName="revoke-keys" Oct 30 05:51:20.087394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.087378 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="efea9f9d-5603-4c46-a2a1-746102dcb82f" containerName="revoke-keys" Oct 30 05:51:20.091248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.091232 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.113366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.113327 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd"] Oct 30 05:51:20.176406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.176367 6171 generic.go:356] "Generic (PLEG): container finished" podID="f70eb699-96ee-43e2-bc52-d3eb63dfe83a" containerID="002c7b7f7177123374bcfec5ba3499fef3367c69d3c39b561c1126b9135b161d" exitCode=0 Oct 30 05:51:20.176569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.176456 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" event={"ID":"f70eb699-96ee-43e2-bc52-d3eb63dfe83a","Type":"ContainerDied","Data":"002c7b7f7177123374bcfec5ba3499fef3367c69d3c39b561c1126b9135b161d"} Oct 30 05:51:20.177941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.177918 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" Oct 30 05:51:20.177941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.177926 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5" event={"ID":"efea9f9d-5603-4c46-a2a1-746102dcb82f","Type":"ContainerDied","Data":"cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701"} Oct 30 05:51:20.178146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.177967 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb409e1cc15a5b3781c1b9d899823d4645b09ca7929c4bed8b1a6fbbd581b701" Oct 30 05:51:20.231789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.231761 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.231991 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.231968 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.232156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.232139 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snjh5\" (UniqueName: \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.338929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.334374 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.338929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.334548 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.338929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.334683 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snjh5\" (UniqueName: \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.338929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.336454 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.352113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.348870 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.366272 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.362756 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snjh5\" (UniqueName: \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\") pod \"maas-api-revoke-sub-d3917d7c-25tgd\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.433066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.432216 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:20.806039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:20.806008 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd"] Oct 30 05:51:21.189097 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.189028 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" event={"ID":"811f8d90-1c5f-4621-a21e-aa77b673230d","Type":"ContainerStarted","Data":"85a6ad44d5872feeb73deffed529aae4353d9f9e472652c9147fd2e95f18fa89"} Oct 30 05:51:21.312093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.311643 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:21.317134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.313823 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:21.317134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.315023 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:51:21.364320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.364296 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:21.466570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.466481 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\" (UniqueName: \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\") pod \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " Oct 30 05:51:21.466570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.466527 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\") pod \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " Oct 30 05:51:21.467113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.466712 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\") pod \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\" (UID: \"f70eb699-96ee-43e2-bc52-d3eb63dfe83a\") " Oct 30 05:51:21.467965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.467904 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca" pod "f70eb699-96ee-43e2-bc52-d3eb63dfe83a" (UID: "f70eb699-96ee-43e2-bc52-d3eb63dfe83a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:21.470463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.470439 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca" pod "f70eb699-96ee-43e2-bc52-d3eb63dfe83a" (UID: "f70eb699-96ee-43e2-bc52-d3eb63dfe83a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:21.470576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.470514 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v" pod "f70eb699-96ee-43e2-bc52-d3eb63dfe83a" (UID: "f70eb699-96ee-43e2-bc52-d3eb63dfe83a"). InnerVolumeSpecName "kube-api-access-7v24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:21.569394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.569362 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v24v\" (UniqueName: \"kubernetes.io/projected/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-kube-api-access-7v24v\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.569394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.569392 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.569394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:21.569401 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f70eb699-96ee-43e2-bc52-d3eb63dfe83a-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:22.202197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:22.202116 6171 generic.go:356] "Generic (PLEG): container finished" podID="811f8d90-1c5f-4621-a21e-aa77b673230d" containerID="2bafb2707c81f700c20eeb4f49c6f84cd5741c8834da0d73a2788e621e038abf" exitCode=0 Oct 30 05:51:22.202347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:22.202199 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" event={"ID":"811f8d90-1c5f-4621-a21e-aa77b673230d","Type":"ContainerDied","Data":"2bafb2707c81f700c20eeb4f49c6f84cd5741c8834da0d73a2788e621e038abf"} Oct 30 05:51:22.203782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:22.203762 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" Oct 30 05:51:22.203892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:22.203783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2" event={"ID":"f70eb699-96ee-43e2-bc52-d3eb63dfe83a","Type":"ContainerDied","Data":"65f16666f28440121e11f853bac0d1911034448b78fc3e0ca9698a5391059ba1"} Oct 30 05:51:22.203892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:22.203820 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f16666f28440121e11f853bac0d1911034448b78fc3e0ca9698a5391059ba1" Oct 30 05:51:23.384233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.384209 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:23.502884 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.502854 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\" (UniqueName: \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\") pod \"811f8d90-1c5f-4621-a21e-aa77b673230d\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " Oct 30 05:51:23.503104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.502893 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\") pod \"811f8d90-1c5f-4621-a21e-aa77b673230d\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " Oct 30 05:51:23.503176 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.503134 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\") pod \"811f8d90-1c5f-4621-a21e-aa77b673230d\" (UID: \"811f8d90-1c5f-4621-a21e-aa77b673230d\") " Oct 30 05:51:23.503545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.503504 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca" pod "811f8d90-1c5f-4621-a21e-aa77b673230d" (UID: "811f8d90-1c5f-4621-a21e-aa77b673230d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:23.503865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.503845 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:23.505252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.505224 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca" pod "811f8d90-1c5f-4621-a21e-aa77b673230d" (UID: "811f8d90-1c5f-4621-a21e-aa77b673230d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:23.505252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.505239 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5" pod "811f8d90-1c5f-4621-a21e-aa77b673230d" (UID: "811f8d90-1c5f-4621-a21e-aa77b673230d"). InnerVolumeSpecName "kube-api-access-snjh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:23.610126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.606562 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snjh5\" (UniqueName: \"kubernetes.io/projected/811f8d90-1c5f-4621-a21e-aa77b673230d-kube-api-access-snjh5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:23.610126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:23.606601 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/811f8d90-1c5f-4621-a21e-aa77b673230d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:24.226567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:24.226530 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" event={"ID":"811f8d90-1c5f-4621-a21e-aa77b673230d","Type":"ContainerDied","Data":"85a6ad44d5872feeb73deffed529aae4353d9f9e472652c9147fd2e95f18fa89"} Oct 30 05:51:24.226765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:24.226576 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85a6ad44d5872feeb73deffed529aae4353d9f9e472652c9147fd2e95f18fa89" Oct 30 05:51:24.226765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:24.226588 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd" Oct 30 05:51:25.278439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.278407 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:51:25.370730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.370703 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f"] Oct 30 05:51:25.371531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371515 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f70eb699-96ee-43e2-bc52-d3eb63dfe83a" containerName="revoke-keys" Oct 30 05:51:25.371586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371534 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f70eb699-96ee-43e2-bc52-d3eb63dfe83a" containerName="revoke-keys" Oct 30 05:51:25.371586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371572 6171 cpu_manager.go:414] "containerMap: removing container" podUID="811f8d90-1c5f-4621-a21e-aa77b673230d" containerName="revoke-keys" Oct 30 05:51:25.371586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371578 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="811f8d90-1c5f-4621-a21e-aa77b673230d" containerName="revoke-keys" Oct 30 05:51:25.371697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371687 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f70eb699-96ee-43e2-bc52-d3eb63dfe83a" containerName="revoke-keys" Oct 30 05:51:25.371743 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.371705 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="811f8d90-1c5f-4621-a21e-aa77b673230d" containerName="revoke-keys" Oct 30 05:51:25.375803 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.375786 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.391842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.391806 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:25.391957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.391875 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:25.396291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.396273 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f"] Oct 30 05:51:25.401481 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.401461 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:25.401602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.401497 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:51:25.540127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.540033 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pzvc\" (UniqueName: \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.540299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.540206 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.540384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.540306 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.641786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.641762 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pzvc\" (UniqueName: \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.641910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.641897 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.642013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.641997 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.642630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.642610 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.644539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.644514 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.664769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.664747 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pzvc\" (UniqueName: \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\") pod \"maas-api-revoke-sub-343574e2-zk87f\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.690920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.690898 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:25.702483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.702460 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6"] Oct 30 05:51:25.709549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.709532 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.734819 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.733160 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6"] Oct 30 05:51:25.794795 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.794736 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4"] Oct 30 05:51:25.801179 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.801158 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:25.820699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.820661 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4"] Oct 30 05:51:25.844930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.844898 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.845267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.845247 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.845446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.845426 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzljk\" (UniqueName: \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.953727 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.953823 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.953911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzljk\" (UniqueName: \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.953939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xpkb\" (UniqueName: \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.954110 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.961574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.954324 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:25.978098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:25.974144 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:26.008143 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.007015 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzljk\" (UniqueName: \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:26.008696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.008668 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-40a76272-5k7p6\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:26.051421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.051341 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:26.078109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.070504 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.078109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.070619 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.078109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.070721 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xpkb\" (UniqueName: \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.092800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.092768 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.117554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.117516 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xpkb\" (UniqueName: \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.154070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.145597 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c3de0bdc-hwfw4\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:26.173298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.173259 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f"] Oct 30 05:51:26.199906 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:26.181936 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804fca7b_2f85_4ac9_a9df_a0e81f9afedd.slice/crio-f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e WatchSource:0}: Error finding container f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e: Status 404 returned error can't find the container with id f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e Oct 30 05:51:26.484079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.482929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" event={"ID":"804fca7b-2f85-4ac9-a9df-a0e81f9afedd","Type":"ContainerStarted","Data":"f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e"} Oct 30 05:51:26.663068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:26.659142 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:27.308138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.299972 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6"] Oct 30 05:51:27.440128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.435145 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.474084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.470483 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.505109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.488870 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.592433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.592363 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" event={"ID":"804fca7b-2f85-4ac9-a9df-a0e81f9afedd","Type":"ContainerStarted","Data":"c662a04bb750cebb0467c7d4f71dafbb752e84a95b81c0576d4f637ba6338836"} Oct 30 05:51:27.662592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.662553 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" event={"ID":"32a9ce81-4778-4883-b74d-e81397c12c58","Type":"ContainerStarted","Data":"fd5863c9bbe73186379badb7e26dccd522ba4a4417b3a58e689826b76e0f69b4"} Oct 30 05:51:27.711641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.711443 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.744085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.734989 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.744085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.735630 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" podStartSLOduration=2.735615003 podStartE2EDuration="2.735615003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:27.661817032 +0000 UTC m=+1146.102194224" watchObservedRunningTime="2026-10-30 05:51:27.735615003 +0000 UTC m=+1146.175992197" Oct 30 05:51:27.764529 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:51:27.763883 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:51:27.769069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.766480 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.777072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.774142 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:27.912514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:27.912476 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.266126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.265180 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.518261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.438127 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.518261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.438839 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:51:28.690075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.687254 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:51:28.751071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.720079 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.751071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.723024 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:51:28.791799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.754782 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.791799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.767144 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.918227 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.926687 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.957201 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.957260 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.957298 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.957331 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:28.965071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.957363 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:51:29.008113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.988622 6171 generic.go:356] "Generic (PLEG): container finished" podID="804fca7b-2f85-4ac9-a9df-a0e81f9afedd" containerID="c662a04bb750cebb0467c7d4f71dafbb752e84a95b81c0576d4f637ba6338836" exitCode=0 Oct 30 05:51:29.140124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:28.995198 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" event={"ID":"804fca7b-2f85-4ac9-a9df-a0e81f9afedd","Type":"ContainerDied","Data":"c662a04bb750cebb0467c7d4f71dafbb752e84a95b81c0576d4f637ba6338836"} Oct 30 05:51:29.297070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.283518 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:29.297070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.283987 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:29.352071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.337446 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" event={"ID":"32a9ce81-4778-4883-b74d-e81397c12c58","Type":"ContainerStarted","Data":"2d7a2c751ea46cecde482ef1a2d1f84320b67561cc22f46ee16d7a6b077910ca"} Oct 30 05:51:29.467123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.439630 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.508697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.504258 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:29.508697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.504317 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.731806 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.735516 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.753533 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.764378 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.769397 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:29.940794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:29.928193 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:30.042068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.041210 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:51:30.188135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.188090 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:30.284978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.284929 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.349348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.349319 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn"] Oct 30 05:51:30.557000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.530670 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.751072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.736165 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:30.772011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.765298 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:30.772011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.769873 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:51:30.988471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.988389 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:51:30.992802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.990196 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn"] Oct 30 05:51:30.992802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.990306 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:51:30.992802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.990362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:51:30.992802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:30.990483 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:31.262115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.254980 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:31.327068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.314997 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:31.687525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.687469 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:31.687783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.687751 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:31.700363 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.700315 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:31.773073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.762536 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:31.802262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.773633 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:31.802262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:31.774523 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:32.019255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.006879 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.019255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.007062 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gml7r\" (UniqueName: \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.019255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.007126 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.070942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.069621 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:32.096094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.096060 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" event={"ID":"4cc81b47-a7d9-4b94-9241-a2f664ffd523","Type":"ContainerStarted","Data":"230446961e2c1c291c8714713aab545156650cc77760b0f007662c7ef882a01a"} Oct 30 05:51:32.096407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.096382 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:32.096636 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.096623 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:51:32.100077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.096808 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:32.100077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.098889 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4"] Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.101626 6171 generic.go:356] "Generic (PLEG): container finished" podID="32a9ce81-4778-4883-b74d-e81397c12c58" containerID="2d7a2c751ea46cecde482ef1a2d1f84320b67561cc22f46ee16d7a6b077910ca" exitCode=0 Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.106269 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" event={"ID":"32a9ce81-4778-4883-b74d-e81397c12c58","Type":"ContainerDied","Data":"2d7a2c751ea46cecde482ef1a2d1f84320b67561cc22f46ee16d7a6b077910ca"} Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.109659 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gml7r\" (UniqueName: \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.109721 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.109928 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.112077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.111645 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.206682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.198784 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.209071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.208114 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gml7r\" (UniqueName: \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\") pod \"maas-api-revoke-sub-0690fdaf-kg8xn\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.219459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.219372 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:32.230349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.230329 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:51:32.252662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.250753 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" podStartSLOduration=7.250735057 podStartE2EDuration="7.250735057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:30.03874552 +0000 UTC m=+1148.479122735" watchObservedRunningTime="2026-10-30 05:51:32.250735057 +0000 UTC m=+1150.691112251" Oct 30 05:51:32.367931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.367902 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:32.379603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.379575 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:51:32.415339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.415274 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:51:32.415480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.415394 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.422162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.422086 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:32.430636 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.430339 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-54d4943b-gw-openshift-default-dockercfg-fdg22\"" Oct 30 05:51:32.524381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.524349 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\") pod \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " Oct 30 05:51:32.524595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.524476 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\" (UniqueName: \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\") pod \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " Oct 30 05:51:32.524732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.524693 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\") pod \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\" (UID: \"804fca7b-2f85-4ac9-a9df-a0e81f9afedd\") " Oct 30 05:51:32.525150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.525113 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca" pod "804fca7b-2f85-4ac9-a9df-a0e81f9afedd" (UID: "804fca7b-2f85-4ac9-a9df-a0e81f9afedd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:32.525646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.525621 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526060 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526020 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526104 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526152 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526364 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526447 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526589 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526631 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mdkz\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526790 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.526894 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526874 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca" pod "804fca7b-2f85-4ac9-a9df-a0e81f9afedd" (UID: "804fca7b-2f85-4ac9-a9df-a0e81f9afedd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:32.527001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.526984 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:32.527723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.527696 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc" pod "804fca7b-2f85-4ac9-a9df-a0e81f9afedd" (UID: "804fca7b-2f85-4ac9-a9df-a0e81f9afedd"). InnerVolumeSpecName "kube-api-access-4pzvc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:32.627937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.627904 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.627985 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628101 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628129 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mdkz\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628240 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628279 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628313 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628341 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628373 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628455 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628529 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:32.628853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628819 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.628853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628838 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.629009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628865 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pzvc\" (UniqueName: \"kubernetes.io/projected/804fca7b-2f85-4ac9-a9df-a0e81f9afedd-kube-api-access-4pzvc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:32.629009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.628987 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.629231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.629209 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.631172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.631147 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.631258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.631220 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.652160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.652134 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.652436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.652416 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mdkz\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\") pod \"e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.761625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.761331 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:32.805968 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:32.805928 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bcf2f3a_6559_42ea_a44e_f0e5abe6bd4f.slice/crio-80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae WatchSource:0}: Error finding container 80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae: Status 404 returned error can't find the container with id 80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae Oct 30 05:51:33.004355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:32.863153 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn"] Oct 30 05:51:33.122133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.121845 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" event={"ID":"804fca7b-2f85-4ac9-a9df-a0e81f9afedd","Type":"ContainerDied","Data":"f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e"} Oct 30 05:51:33.122133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.121909 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9e69b25f17a65ae20bb5aeda32e819320d7c864830dfb7102111093cba4040e" Oct 30 05:51:33.122133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.121973 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f" Oct 30 05:51:33.125037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.124915 6171 generic.go:356] "Generic (PLEG): container finished" podID="4cc81b47-a7d9-4b94-9241-a2f664ffd523" containerID="b0f17496730c18917520c0e4268a3439eb3f80e03c2adacbf2b87b34b7d8e28d" exitCode=0 Oct 30 05:51:33.125037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.124998 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" event={"ID":"4cc81b47-a7d9-4b94-9241-a2f664ffd523","Type":"ContainerDied","Data":"b0f17496730c18917520c0e4268a3439eb3f80e03c2adacbf2b87b34b7d8e28d"} Oct 30 05:51:33.127697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.127672 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" event={"ID":"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f","Type":"ContainerStarted","Data":"80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae"} Oct 30 05:51:33.333771 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:33.333739 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod927271a8_8f85_454c_a5a4_56206980d6c4.slice/crio-0d795481aa2a5144404b9309b2b6eb404c87c4c202cfddf317d6892dc67b033e WatchSource:0}: Error finding container 0d795481aa2a5144404b9309b2b6eb404c87c4c202cfddf317d6892dc67b033e: Status 404 returned error can't find the container with id 0d795481aa2a5144404b9309b2b6eb404c87c4c202cfddf317d6892dc67b033e Oct 30 05:51:33.336782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.336749 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:33.336892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.336820 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:33.340302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.339513 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:51:33.345584 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.343329 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:33.364429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.364410 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:33.443103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.443081 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\") pod \"32a9ce81-4778-4883-b74d-e81397c12c58\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " Oct 30 05:51:33.443225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.443134 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\") pod \"32a9ce81-4778-4883-b74d-e81397c12c58\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " Oct 30 05:51:33.443285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.443230 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\" (UniqueName: \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\") pod \"32a9ce81-4778-4883-b74d-e81397c12c58\" (UID: \"32a9ce81-4778-4883-b74d-e81397c12c58\") " Oct 30 05:51:33.443432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.443406 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca" pod "32a9ce81-4778-4883-b74d-e81397c12c58" (UID: "32a9ce81-4778-4883-b74d-e81397c12c58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:33.444248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.444222 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.445287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.445258 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca" pod "32a9ce81-4778-4883-b74d-e81397c12c58" (UID: "32a9ce81-4778-4883-b74d-e81397c12c58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:33.445451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.445429 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk" pod "32a9ce81-4778-4883-b74d-e81397c12c58" (UID: "32a9ce81-4778-4883-b74d-e81397c12c58"). InnerVolumeSpecName "kube-api-access-xzljk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:33.545993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.545961 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32a9ce81-4778-4883-b74d-e81397c12c58-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:33.545993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:33.545993 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzljk\" (UniqueName: \"kubernetes.io/projected/32a9ce81-4778-4883-b74d-e81397c12c58-kube-api-access-xzljk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:34.138407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.138378 6171 generic.go:356] "Generic (PLEG): container finished" podID="0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" containerID="c73a2560163edf94458004fbc62d15593b7eb81b5e967fa7b402eb0373b2e048" exitCode=0 Oct 30 05:51:34.138780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.138457 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" event={"ID":"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f","Type":"ContainerDied","Data":"c73a2560163edf94458004fbc62d15593b7eb81b5e967fa7b402eb0373b2e048"} Oct 30 05:51:34.139894 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.139871 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" Oct 30 05:51:34.140018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.139912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6" event={"ID":"32a9ce81-4778-4883-b74d-e81397c12c58","Type":"ContainerDied","Data":"fd5863c9bbe73186379badb7e26dccd522ba4a4417b3a58e689826b76e0f69b4"} Oct 30 05:51:34.140018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.139963 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5863c9bbe73186379badb7e26dccd522ba4a4417b3a58e689826b76e0f69b4" Oct 30 05:51:34.141898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.141873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" event={"ID":"927271a8-8f85-454c-a5a4-56206980d6c4","Type":"ContainerStarted","Data":"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5"} Oct 30 05:51:34.142017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.141924 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" event={"ID":"927271a8-8f85-454c-a5a4-56206980d6c4","Type":"ContainerStarted","Data":"0d795481aa2a5144404b9309b2b6eb404c87c4c202cfddf317d6892dc67b033e"} Oct 30 05:51:34.455240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.356766 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" podStartSLOduration=2.356745353 podStartE2EDuration="2.356745353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:34.248553877 +0000 UTC m=+1152.688931070" watchObservedRunningTime="2026-10-30 05:51:34.356745353 +0000 UTC m=+1152.797122545" Oct 30 05:51:34.763067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.761839 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:34.774661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.774295 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:34.898920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.898883 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:34.956829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.956795 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\") pod \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " Oct 30 05:51:34.957204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.957185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\" (UniqueName: \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\") pod \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " Oct 30 05:51:34.957273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.957236 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\") pod \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\" (UID: \"4cc81b47-a7d9-4b94-9241-a2f664ffd523\") " Oct 30 05:51:34.957833 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.957800 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca" pod "4cc81b47-a7d9-4b94-9241-a2f664ffd523" (UID: "4cc81b47-a7d9-4b94-9241-a2f664ffd523"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:34.959081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.959059 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca" pod "4cc81b47-a7d9-4b94-9241-a2f664ffd523" (UID: "4cc81b47-a7d9-4b94-9241-a2f664ffd523"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:34.959212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:34.959180 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb" pod "4cc81b47-a7d9-4b94-9241-a2f664ffd523" (UID: "4cc81b47-a7d9-4b94-9241-a2f664ffd523"). InnerVolumeSpecName "kube-api-access-8xpkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:35.059554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.059524 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xpkb\" (UniqueName: \"kubernetes.io/projected/4cc81b47-a7d9-4b94-9241-a2f664ffd523-kube-api-access-8xpkb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:35.059554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.059551 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:35.059714 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.059560 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc81b47-a7d9-4b94-9241-a2f664ffd523-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:35.156842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.156810 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" Oct 30 05:51:35.156842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.156834 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4" event={"ID":"4cc81b47-a7d9-4b94-9241-a2f664ffd523","Type":"ContainerDied","Data":"230446961e2c1c291c8714713aab545156650cc77760b0f007662c7ef882a01a"} Oct 30 05:51:35.157361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.156904 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="230446961e2c1c291c8714713aab545156650cc77760b0f007662c7ef882a01a" Oct 30 05:51:35.157361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.157333 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:35.158502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.158482 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:51:35.370348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.370322 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:35.464297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.464268 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\" (UniqueName: \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\") pod \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " Oct 30 05:51:35.464454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.464430 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\") pod \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " Oct 30 05:51:35.464515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.464503 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\") pod \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\" (UID: \"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f\") " Oct 30 05:51:35.464753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.464729 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca" pod "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" (UID: "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:35.465079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.465037 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:35.466448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.466426 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r" pod "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" (UID: "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f"). InnerVolumeSpecName "kube-api-access-gml7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:35.466538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.466516 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca" pod "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" (UID: "0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:35.566634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.566610 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gml7r\" (UniqueName: \"kubernetes.io/projected/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-kube-api-access-gml7r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:35.566634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:35.566632 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:36.047598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.047563 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:51:36.048348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048331 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4cc81b47-a7d9-4b94-9241-a2f664ffd523" containerName="revoke-keys" Oct 30 05:51:36.048410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048350 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc81b47-a7d9-4b94-9241-a2f664ffd523" containerName="revoke-keys" Oct 30 05:51:36.048410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048364 6171 cpu_manager.go:414] "containerMap: removing container" podUID="32a9ce81-4778-4883-b74d-e81397c12c58" containerName="revoke-keys" Oct 30 05:51:36.048410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048369 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a9ce81-4778-4883-b74d-e81397c12c58" containerName="revoke-keys" Oct 30 05:51:36.048410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048379 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" containerName="revoke-keys" Oct 30 05:51:36.048410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048385 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" containerName="revoke-keys" Oct 30 05:51:36.048544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048430 6171 cpu_manager.go:414] "containerMap: removing container" podUID="804fca7b-2f85-4ac9-a9df-a0e81f9afedd" containerName="revoke-keys" Oct 30 05:51:36.048544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048436 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804fca7b-2f85-4ac9-a9df-a0e81f9afedd" containerName="revoke-keys" Oct 30 05:51:36.048617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048606 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="804fca7b-2f85-4ac9-a9df-a0e81f9afedd" containerName="revoke-keys" Oct 30 05:51:36.048649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048628 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc81b47-a7d9-4b94-9241-a2f664ffd523" containerName="revoke-keys" Oct 30 05:51:36.048649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048635 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a9ce81-4778-4883-b74d-e81397c12c58" containerName="revoke-keys" Oct 30 05:51:36.048649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.048642 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" containerName="revoke-keys" Oct 30 05:51:36.053098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.053039 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.096142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.096112 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:51:36.100500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.100468 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:51:36.107645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.107605 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.118017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.117991 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:51:36.173174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.173139 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" Oct 30 05:51:36.173578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.173146 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn" event={"ID":"0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f","Type":"ContainerDied","Data":"80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae"} Oct 30 05:51:36.173578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.173306 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80be7479c4731b41112e06ab3c72c2529983487c3cd7f7bb4846077d3293b7ae" Oct 30 05:51:36.191739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191707 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.191891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191752 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.191891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191775 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.191891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191878 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191917 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.191939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192088 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsmlx\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192173 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192200 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htdl8\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192300 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192337 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192371 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192431 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192469 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192568 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192516 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.192700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192684 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.192752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.192716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.293746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293720 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.293873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293756 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.293873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293789 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.293873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293820 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.293873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293836 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293943 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.293979 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294107 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294158 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294184 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294266 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294310 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294359 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294344 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294376 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsmlx\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294397 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294419 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294535 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294638 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294678 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294701 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.294805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294423 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294774 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htdl8\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.294982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.294875 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.295186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.295162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.295316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.295249 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.295316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.295275 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.295794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.295770 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.296109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.296086 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.297124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.297102 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.297124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.297124 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.297289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.297273 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.297348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.297331 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.316897 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.316841 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.317002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.316897 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.317605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.317583 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htdl8\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.317874 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.317857 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsmlx\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.368024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.368000 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:36.425522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.425486 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:36.713577 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:36.713551 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30edac37_a8dd_40c5_bccd_53db5d8fd7cd.slice/crio-dd80a42c840c39a282a44239976e17c1d4c7d55ed0da04f64f5c69bfac9889df WatchSource:0}: Error finding container dd80a42c840c39a282a44239976e17c1d4c7d55ed0da04f64f5c69bfac9889df: Status 404 returned error can't find the container with id dd80a42c840c39a282a44239976e17c1d4c7d55ed0da04f64f5c69bfac9889df Oct 30 05:51:36.716483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.716439 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.716609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.716529 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.716609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.716597 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.718148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.718124 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:51:36.728192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.728170 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:51:36.734869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.734831 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.764423 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.762525 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:51:36.800944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.797554 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.800944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.797650 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.800944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.797696 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:36.805277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.805235 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:51:36.807315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807263 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807419 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807333 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807419 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807370 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807552 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807589 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807769 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtgw5\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807847 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.807936 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807889 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.808017 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.807999 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.909846 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.909905 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.909969 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.910009 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911210 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911219 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911360 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911547 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtgw5\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911628 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.911675 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.912136 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.912489 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.912180 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.913533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.912813 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.913983 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.913952 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.914363 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.914344 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.937576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.936183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:36.938566 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:36.938543 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtgw5\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\") pod \"e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:37.091190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.091159 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:37.210248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.210200 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" event={"ID":"30edac37-a8dd-40c5-bccd-53db5d8fd7cd","Type":"ContainerStarted","Data":"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27"} Oct 30 05:51:37.210667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.210278 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" event={"ID":"30edac37-a8dd-40c5-bccd-53db5d8fd7cd","Type":"ContainerStarted","Data":"dd80a42c840c39a282a44239976e17c1d4c7d55ed0da04f64f5c69bfac9889df"} Oct 30 05:51:37.212274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.212246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" event={"ID":"2afa0d1a-5836-4c90-8af6-5572f0979b9f","Type":"ContainerStarted","Data":"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90"} Oct 30 05:51:37.212436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.212291 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" event={"ID":"2afa0d1a-5836-4c90-8af6-5572f0979b9f","Type":"ContainerStarted","Data":"1a277da59d0b19bb9dd3cd3d402d6f76a90c52d5fde9082ab9f215dddbe64d0f"} Oct 30 05:51:37.262255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.262194 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podStartSLOduration=1.262175013 podStartE2EDuration="1.262175013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:37.259183186 +0000 UTC m=+1155.699560379" watchObservedRunningTime="2026-10-30 05:51:37.262175013 +0000 UTC m=+1155.702552205" Oct 30 05:51:37.329617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.329556 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podStartSLOduration=1.32953772 podStartE2EDuration="1.32953772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:37.306825113 +0000 UTC m=+1155.747202306" watchObservedRunningTime="2026-10-30 05:51:37.32953772 +0000 UTC m=+1155.769914912" Oct 30 05:51:37.373421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.373351 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:37.374732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.374703 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:37.374884 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.374762 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:37.456517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.456478 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:37.460382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.460352 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:37.460482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.460414 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:37.462545 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:37.462518 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048f2d6f_f420_4e1e_8a9e_b922c5b785dc.slice/crio-a2064406831e3f4b90d8b7b6456908a324e7634e5fff298a93263dc1baabd0a0 WatchSource:0}: Error finding container a2064406831e3f4b90d8b7b6456908a324e7634e5fff298a93263dc1baabd0a0: Status 404 returned error can't find the container with id a2064406831e3f4b90d8b7b6456908a324e7634e5fff298a93263dc1baabd0a0 Oct 30 05:51:37.465339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.465221 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.465806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.465305 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.465806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.465474 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:37.467400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.467379 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:51:37.661379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.659102 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:51:37.747180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.746274 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.767843 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.767813 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-54d4943b-dockercfg-dgx5g\"" Oct 30 05:51:37.768137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.768088 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-54d4943b\"" Oct 30 05:51:37.837348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.815730 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:51:37.868283 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.868168 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6q99\" (UniqueName: \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.869521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.868705 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.947508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.947465 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:51:37.969982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.969944 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6q99\" (UniqueName: \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.970252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.970227 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.971213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.971184 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:37.971444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.971427 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:51:37.971592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:37.971571 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.073069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.069781 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6q99\" (UniqueName: \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\") pod \"payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:38.108258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.074354 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfjt\" (UniqueName: \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.108917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.108886 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.241453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.241412 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfjt\" (UniqueName: \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.256856 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.242454 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.257453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.246011 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.257982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.257946 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:38.402739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.389373 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:38.402739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.389442 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:38.405365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.405332 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfjt\" (UniqueName: \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\") pod \"payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:38.488358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.488318 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:38.488580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.488410 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:38.546938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.546889 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" event={"ID":"048f2d6f-f420-4e1e-8a9e-b922c5b785dc","Type":"ContainerStarted","Data":"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd"} Oct 30 05:51:38.547500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.547355 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" event={"ID":"048f2d6f-f420-4e1e-8a9e-b922c5b785dc","Type":"ContainerStarted","Data":"a2064406831e3f4b90d8b7b6456908a324e7634e5fff298a93263dc1baabd0a0"} Oct 30 05:51:38.654082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.653999 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podStartSLOduration=2.653979229 podStartE2EDuration="2.653979229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:38.616010191 +0000 UTC m=+1157.056387384" watchObservedRunningTime="2026-10-30 05:51:38.653979229 +0000 UTC m=+1157.094356422" Oct 30 05:51:38.659582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:38.659558 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:39.109630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.109596 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:39.156595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.156561 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:39.156840 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.156809 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:39.270109 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:39.267374 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2777af9_55f2_492c_ae04_615c9827aa22.slice/crio-0d88fadccf39c31268a620de92a2de2fe6a10c4c0fa297b1f92c211ea52da3fb WatchSource:0}: Error finding container 0d88fadccf39c31268a620de92a2de2fe6a10c4c0fa297b1f92c211ea52da3fb: Status 404 returned error can't find the container with id 0d88fadccf39c31268a620de92a2de2fe6a10c4c0fa297b1f92c211ea52da3fb Oct 30 05:51:39.276662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.275464 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:51:39.377319 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.377013 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:39.377319 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.377088 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:39.379013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.378985 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:51:39.439025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.438993 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:39.466763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.463541 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:39.578514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.578481 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" event={"ID":"405c2936-8199-42f0-b09f-3878abb8a351","Type":"ContainerStarted","Data":"8cd865f5b90bdbb74e37ef442bd50ab2a30378fb9e6e3c160a7be16a4b44d2da"} Oct 30 05:51:39.579861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:39.579830 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" event={"ID":"d2777af9-55f2-492c-ae04-615c9827aa22","Type":"ContainerStarted","Data":"0d88fadccf39c31268a620de92a2de2fe6a10c4c0fa297b1f92c211ea52da3fb"} Oct 30 05:51:40.196124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.112479 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:40.196124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.112542 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:40.196124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.128090 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562"] Oct 30 05:51:40.592072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.591762 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:40.592072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.591821 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:40.804195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.804160 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:40.868209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:40.833263 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:41.007323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.007289 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562"] Oct 30 05:51:41.007621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.007601 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:41.007855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.007831 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" event={"ID":"d2777af9-55f2-492c-ae04-615c9827aa22","Type":"ContainerStarted","Data":"1767376e0e9941808fc6ba2180aa4bc9a346e2ad6952af312d98b776365a0d58"} Oct 30 05:51:41.008118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.008100 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.040812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.027856 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:41.040812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.028006 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:51:41.040812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.028106 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:41.040812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.028217 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:41.054349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.054322 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" event={"ID":"405c2936-8199-42f0-b09f-3878abb8a351","Type":"ContainerStarted","Data":"ef2b4fe2238d621f216e747caf9bcae3344159f47cd5373cf8b679868bf6eb51"} Oct 30 05:51:41.068334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.068285 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" podStartSLOduration=4.068250536 podStartE2EDuration="4.068250536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:41.061833958 +0000 UTC m=+1159.502211144" watchObservedRunningTime="2026-10-30 05:51:41.068250536 +0000 UTC m=+1159.508627729" Oct 30 05:51:41.070152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.070127 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:41.091074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.080330 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.091074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.080557 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjxnh\" (UniqueName: \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.091074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.080732 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.102415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.102381 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:41.102673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.102646 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:41.163206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.155486 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" podStartSLOduration=4.155463169 podStartE2EDuration="4.155463169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:41.118511522 +0000 UTC m=+1159.558888716" watchObservedRunningTime="2026-10-30 05:51:41.155463169 +0000 UTC m=+1159.595840362" Oct 30 05:51:41.212071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.201536 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjxnh\" (UniqueName: \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.212071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.201740 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.212071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.201921 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.212071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.204198 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.257065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.256780 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.295938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.295902 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjxnh\" (UniqueName: \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\") pod \"maas-api-revoke-sub-e66adadd-ql562\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.371505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.371464 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:41.372573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.372530 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:41.424391 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.424303 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:41.429450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.429414 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:41.429808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:41.429777 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:42.117250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.107075 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:42.117250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.107137 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:42.372525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.372485 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:42.397182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.397130 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:42.525830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.525253 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:42.525830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.525319 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:42.774109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.770079 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562"] Oct 30 05:51:42.774109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.773291 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:42.774486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:42.774462 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:43.101626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.101169 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:43.101626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.101232 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:43.369721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.369634 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:43.369721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.369686 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:43.444070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.440887 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:43.444070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.440941 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:43.769839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.769793 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" event={"ID":"1ec6a329-44fe-40da-b9c9-21854045a52c","Type":"ContainerStarted","Data":"2259620fc6c0bfdb76f76ef1f964d09b50b620511a1237c863af656b8b0df419"} Oct 30 05:51:43.770032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.769851 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" event={"ID":"1ec6a329-44fe-40da-b9c9-21854045a52c","Type":"ContainerStarted","Data":"59bcdec73d57e8f671552d091460d23ce33c7735ad596c16f167270dd29abeb9"} Oct 30 05:51:43.864296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:43.864238 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" podStartSLOduration=4.864219598 podStartE2EDuration="4.864219598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:43.825518319 +0000 UTC m=+1162.265895512" watchObservedRunningTime="2026-10-30 05:51:43.864219598 +0000 UTC m=+1162.304596790" Oct 30 05:51:44.099130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.097122 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:44.099130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.097185 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:44.368861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.368765 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body= Oct 30 05:51:44.368861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.368834 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" Oct 30 05:51:44.426184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.426142 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:44.426571 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.426209 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:44.793571 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.793533 6171 generic.go:356] "Generic (PLEG): container finished" podID="1ec6a329-44fe-40da-b9c9-21854045a52c" containerID="2259620fc6c0bfdb76f76ef1f964d09b50b620511a1237c863af656b8b0df419" exitCode=0 Oct 30 05:51:44.794067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:44.794022 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" event={"ID":"1ec6a329-44fe-40da-b9c9-21854045a52c","Type":"ContainerDied","Data":"2259620fc6c0bfdb76f76ef1f964d09b50b620511a1237c863af656b8b0df419"} Oct 30 05:51:45.104134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.095512 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:45.104134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.095601 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:45.228327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.224828 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng"] Oct 30 05:51:45.237078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.236353 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.240928 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.240589 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng"] Oct 30 05:51:45.360000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.359909 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsw7g\" (UniqueName: \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.360168 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.360127 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.360208 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.360193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.426235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.426200 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body= Oct 30 05:51:45.426732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.426256 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" Oct 30 05:51:45.461957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.461917 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsw7g\" (UniqueName: \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.462209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.462066 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.462209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.462124 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.463105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.463027 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.465258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.465230 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.486279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.486253 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsw7g\" (UniqueName: \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\") pod \"maas-api-revoke-sub-325a4b2e-q7jng\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.559453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.559401 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:45.978108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:45.976517 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng"] Oct 30 05:51:46.115727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.095081 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" start-of-body= Oct 30 05:51:46.115727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.095150 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.144:15021/healthz/ready\": dial tcp 10.133.0.144:15021: connect: connection refused" Oct 30 05:51:46.290231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.290202 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:46.306117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.306091 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:46.308239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.308216 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:46.322431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.322394 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:51:46.345092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.345059 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\") pod \"1ec6a329-44fe-40da-b9c9-21854045a52c\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " Oct 30 05:51:46.345222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.345138 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\" (UniqueName: \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\") pod \"1ec6a329-44fe-40da-b9c9-21854045a52c\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " Oct 30 05:51:46.345265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.345226 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\") pod \"1ec6a329-44fe-40da-b9c9-21854045a52c\" (UID: \"1ec6a329-44fe-40da-b9c9-21854045a52c\") " Oct 30 05:51:46.346510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.346488 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca" pod "1ec6a329-44fe-40da-b9c9-21854045a52c" (UID: "1ec6a329-44fe-40da-b9c9-21854045a52c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:46.374508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.356114 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh" pod "1ec6a329-44fe-40da-b9c9-21854045a52c" (UID: "1ec6a329-44fe-40da-b9c9-21854045a52c"). InnerVolumeSpecName "kube-api-access-vjxnh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:46.374508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.370554 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca" pod "1ec6a329-44fe-40da-b9c9-21854045a52c" (UID: "1ec6a329-44fe-40da-b9c9-21854045a52c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:46.449957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.449924 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:46.450487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.450470 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjxnh\" (UniqueName: \"kubernetes.io/projected/1ec6a329-44fe-40da-b9c9-21854045a52c-kube-api-access-vjxnh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:46.450614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.450602 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ec6a329-44fe-40da-b9c9-21854045a52c-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:46.623178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.622456 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:46.623492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.623460 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:46.624379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.624358 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:51:46.894585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.894543 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" event={"ID":"1ec6a329-44fe-40da-b9c9-21854045a52c","Type":"ContainerDied","Data":"59bcdec73d57e8f671552d091460d23ce33c7735ad596c16f167270dd29abeb9"} Oct 30 05:51:46.894787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.894608 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59bcdec73d57e8f671552d091460d23ce33c7735ad596c16f167270dd29abeb9" Oct 30 05:51:46.894787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.894675 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562" Oct 30 05:51:46.913271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.913164 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" event={"ID":"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0","Type":"ContainerStarted","Data":"efc32d916f532891f80f2780aeb44a84a4023a2de7503543141a7e1458ea107c"} Oct 30 05:51:46.913415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.913284 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" event={"ID":"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0","Type":"ContainerStarted","Data":"46e052c0e4149a6284aa4982204edee840e618403e568c73edc89365cbca13f8"} Oct 30 05:51:46.997699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:46.997645 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" podStartSLOduration=1.997630732 podStartE2EDuration="1.997630732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:46.964354293 +0000 UTC m=+1165.404731488" watchObservedRunningTime="2026-10-30 05:51:46.997630732 +0000 UTC m=+1165.438007934" Oct 30 05:51:47.333120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:47.333086 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:47.333650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:47.333625 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:47.334445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:47.334425 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:51:47.924344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:47.924117 6171 generic.go:356] "Generic (PLEG): container finished" podID="c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" containerID="efc32d916f532891f80f2780aeb44a84a4023a2de7503543141a7e1458ea107c" exitCode=0 Oct 30 05:51:47.925147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:47.925127 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" event={"ID":"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0","Type":"ContainerDied","Data":"efc32d916f532891f80f2780aeb44a84a4023a2de7503543141a7e1458ea107c"} Oct 30 05:51:49.105657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.105624 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:49.128349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.128321 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\") pod \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " Oct 30 05:51:49.128501 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.128488 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\" (UniqueName: \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\") pod \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " Oct 30 05:51:49.128601 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.128589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\") pod \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\" (UID: \"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0\") " Oct 30 05:51:49.128705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.128683 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca" pod "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" (UID: "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:49.129436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.129404 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:49.130826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.130795 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca" pod "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" (UID: "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:49.130920 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.130878 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g" pod "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" (UID: "c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0"). InnerVolumeSpecName "kube-api-access-tsw7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:49.231573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.231545 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsw7g\" (UniqueName: \"kubernetes.io/projected/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-kube-api-access-tsw7g\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:49.231573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.231573 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:49.949572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.949494 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" Oct 30 05:51:49.949572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.949524 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng" event={"ID":"c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0","Type":"ContainerDied","Data":"46e052c0e4149a6284aa4982204edee840e618403e568c73edc89365cbca13f8"} Oct 30 05:51:49.949737 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:49.949575 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46e052c0e4149a6284aa4982204edee840e618403e568c73edc89365cbca13f8" Oct 30 05:51:51.126291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.126256 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:51:51.127260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127238 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1ec6a329-44fe-40da-b9c9-21854045a52c" containerName="revoke-keys" Oct 30 05:51:51.127393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127259 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ec6a329-44fe-40da-b9c9-21854045a52c" containerName="revoke-keys" Oct 30 05:51:51.127393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127284 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" containerName="revoke-keys" Oct 30 05:51:51.127393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127291 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" containerName="revoke-keys" Oct 30 05:51:51.127545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127466 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ec6a329-44fe-40da-b9c9-21854045a52c" containerName="revoke-keys" Oct 30 05:51:51.127545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.127481 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" containerName="revoke-keys" Oct 30 05:51:51.131802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.131781 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.159115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.159087 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:51:51.166451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166425 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166467 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166488 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166606 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166743 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166810 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166799 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166823 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.166891 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.166867 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.379123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.378962 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn"] Oct 30 05:51:51.394948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.394907 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.439343 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.439410 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.439513 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.439613 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.439701 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.440032 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.440097 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.441084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.440130 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.445156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.444151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.461086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.448738 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.461086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.449462 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.461086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.450399 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.461086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.450677 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.461086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.450956 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.468712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.462227 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn"] Oct 30 05:51:51.468712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.462358 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.472422 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.472391 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.473718 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.473695 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.477398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.477363 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\") pod \"e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.479401 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.479358 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:51.479491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.479409 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:51:51.479567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.479524 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:51.480031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.479673 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:51.647318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.647236 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.647318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.647285 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.647586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.647561 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.748678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.748646 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.748848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.748701 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.749010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.748985 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.749389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.749362 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.751337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.751311 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.752321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.752305 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:51.772614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.772585 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") pod \"maas-api-revoke-sub-e4325a71-9tzsn\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:51.837069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:51.828406 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:52.146468 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:52.146438 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0370a87b_2655_40d6_9c50_018d44049314.slice/crio-4999bb95125fc7d719063b0fca6b8c39a575ff6ffd151a2558ed53abbc96be90 WatchSource:0}: Error finding container 4999bb95125fc7d719063b0fca6b8c39a575ff6ffd151a2558ed53abbc96be90: Status 404 returned error can't find the container with id 4999bb95125fc7d719063b0fca6b8c39a575ff6ffd151a2558ed53abbc96be90 Oct 30 05:51:52.149131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.149097 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:52.149229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.149167 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:52.149229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.149209 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:51:52.150846 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.150824 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:51:52.390099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.387413 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn"] Oct 30 05:51:52.670717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:52.670643 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:51:53.015978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.015166 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" event={"ID":"0370a87b-2655-40d6-9c50-018d44049314","Type":"ContainerStarted","Data":"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66"} Oct 30 05:51:53.015978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.015268 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" event={"ID":"0370a87b-2655-40d6-9c50-018d44049314","Type":"ContainerStarted","Data":"4999bb95125fc7d719063b0fca6b8c39a575ff6ffd151a2558ed53abbc96be90"} Oct 30 05:51:53.030200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.028080 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" event={"ID":"f5a57c41-036a-4f5c-a7db-6f0ded6af417","Type":"ContainerStarted","Data":"b89fca9d05fdbd70efd172da3cd5cdb222b3cac4fd8bc806362a49b47f644680"} Oct 30 05:51:53.107070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.101233 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podStartSLOduration=2.101202839 podStartE2EDuration="2.101202839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:53.078684703 +0000 UTC m=+1171.519061897" watchObservedRunningTime="2026-10-30 05:51:53.101202839 +0000 UTC m=+1171.541580032" Oct 30 05:51:53.752814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.752781 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:51:53.753958 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.753937 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:53.754086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.753980 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:53.772855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:53.772836 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:51:54.039910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:54.039873 6171 generic.go:356] "Generic (PLEG): container finished" podID="f5a57c41-036a-4f5c-a7db-6f0ded6af417" containerID="2e2ca50881b77f08ecdcf7d398689c6e741503f81ccf8392c68e3f04acbba538" exitCode=0 Oct 30 05:51:54.040127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:54.039920 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" event={"ID":"f5a57c41-036a-4f5c-a7db-6f0ded6af417","Type":"ContainerDied","Data":"2e2ca50881b77f08ecdcf7d398689c6e741503f81ccf8392c68e3f04acbba538"} Oct 30 05:51:54.798069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:54.797344 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:54.798069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:54.797402 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:55.214121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.206444 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.226222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.225626 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.515070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.514146 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.847368 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.847914 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.848298 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.892707 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.892766 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:55.929125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.918501 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.037077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:55.969614 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:56.149070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.144202 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.269880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.269839 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:56.271780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.271747 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:56.529231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.529126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.835351 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.835450 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.847699 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.848520 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.849542 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:51:56.858121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.854265 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:51:56.859231 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.181s" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859339 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859479 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859537 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859588 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859643 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859697 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:51:56.913799 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.859752 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:51:56.963410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:56.963378 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:51:57.156144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.152382 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:57.156144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.155303 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:51:57.275711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.275673 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2"] Oct 30 05:51:57.611306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.611273 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:57.711258 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.710720 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-7gh9t" podUID="9c5b4302-435f-4bc0-8578-73fa12bdb419" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": context deadline exceeded" Oct 30 05:51:57.755944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.755886 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.791123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.791075 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:57.791440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.791390 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:57.848207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.848168 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:57.902661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.849854 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:57.911259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.849887 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:57.988683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:57.873751 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.021281 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.017207 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:58.119642 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:51:58.119598 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:51:58.153129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.152997 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:58.251072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.212213 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\") pod \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " Oct 30 05:51:58.251072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.212437 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") pod \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " Oct 30 05:51:58.251072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.212563 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\") pod \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " Oct 30 05:51:58.521217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.521167 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:51:58.677449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.677411 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:51:58.777114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.773251 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca" pod "f5a57c41-036a-4f5c-a7db-6f0ded6af417" (UID: "f5a57c41-036a-4f5c-a7db-6f0ded6af417"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:58.794801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.794758 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq" pod "f5a57c41-036a-4f5c-a7db-6f0ded6af417" (UID: "f5a57c41-036a-4f5c-a7db-6f0ded6af417"). InnerVolumeSpecName "kube-api-access-vwrkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:58.831184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.826402 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca" pod "f5a57c41-036a-4f5c-a7db-6f0ded6af417" (UID: "f5a57c41-036a-4f5c-a7db-6f0ded6af417"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.887445 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.887504 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.938237 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.938683 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.939009 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:51:58.958098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:58.939377 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.027804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.027709 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:51:59.070352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.070318 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") pod \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\" (UID: \"f5a57c41-036a-4f5c-a7db-6f0ded6af417\") " Oct 30 05:51:59.095372 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:51:59.095343 6171 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/f5a57c41-036a-4f5c-a7db-6f0ded6af417/volumes/kubernetes.io~projected/kube-api-access-vwrkq Oct 30 05:51:59.095705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.095680 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq" pod "f5a57c41-036a-4f5c-a7db-6f0ded6af417" (UID: "f5a57c41-036a-4f5c-a7db-6f0ded6af417"). InnerVolumeSpecName "kube-api-access-vwrkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:59.147626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.114357 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" Oct 30 05:51:59.169268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.169207 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:51:59.245865 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:51:59.245829 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.572s" Oct 30 05:51:59.246182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.246163 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:51:59.246439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.246416 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn" event={"ID":"f5a57c41-036a-4f5c-a7db-6f0ded6af417","Type":"ContainerDied","Data":"b89fca9d05fdbd70efd172da3cd5cdb222b3cac4fd8bc806362a49b47f644680"} Oct 30 05:51:59.247827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.246557 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b89fca9d05fdbd70efd172da3cd5cdb222b3cac4fd8bc806362a49b47f644680" Oct 30 05:51:59.261373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.261342 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c8360874-ph9p2"] Oct 30 05:51:59.261678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.261660 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:59.438524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.438439 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5a57c41-036a-4f5c-a7db-6f0ded6af417-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:59.543340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.543304 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:51:59.543660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.543609 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.593586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.593552 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwrkq\" (UniqueName: \"kubernetes.io/projected/f5a57c41-036a-4f5c-a7db-6f0ded6af417-kube-api-access-vwrkq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:51:59.593887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.593853 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.601068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.597480 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:51:59.867691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.814271 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:51:59.867691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.814330 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:51:59.867691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.814406 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:51:59.953362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:51:59.953297 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.090320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.047472 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.290339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.052791 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:00.298501 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.054441 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:00.299023 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.054487 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:00.435871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.055214 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.452117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.093648 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:52:00.594285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.594239 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:00.622099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.621915 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e8e42ef-89a8-47d7-a6ae-350e4f044b30" path="/var/lib/kubelet/pods/5e8e42ef-89a8-47d7-a6ae-350e4f044b30/volumes" Oct 30 05:52:00.681136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.660439 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:52:00.681461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.677885 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" podUID="3e622128-5588-4764-91dd-f7759834b7eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:00.825962 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:52:00.746393 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.065s" Oct 30 05:52:00.825962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.746553 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:52:00.825962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.770517 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:00.825962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.770568 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:00.997279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:00.993550 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.337684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.326379 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:01.337684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.326805 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:52:01.383692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.357118 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:01.433291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.432751 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:01.433291 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.432818 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.109:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.515071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.514450 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:01.515071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.514809 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:01.543274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.543235 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:01.547155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.547113 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.756173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.715082 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.756173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.715670 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:01.756173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.715706 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:01.760935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.760888 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:01.761228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.761191 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:01.788109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.762380 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:02.080193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:01.996796 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:52:02.080193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.017151 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:02.080193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.017222 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:02.080193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.060426 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:02.390128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.349524 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:02.586096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.385256 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:52:02.586096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.410507 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:02.586096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.533143 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:02.586096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.533195 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:02.936596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.936551 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:02.988068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.944136 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:52:02.988068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.945643 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 05:52:02.997115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:02.997075 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:03.247491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.222216 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:52:03.247491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.222282 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:03.305070 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:52:03.289894 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.575s" Oct 30 05:52:03.395071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.337811 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:03.459248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.459206 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:03.867912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.867871 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:03.911701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:03.911639 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:04.340338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.340227 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:04.341239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.341208 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:04.341808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.341784 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:04.349605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.349267 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:04.372523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.350857 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:04.372523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.369797 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:52:04.372523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.370862 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:04.372523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.371854 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:04.772066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.769130 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:04.772066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.769192 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:04.817226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:04.815719 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:52:05.397439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.397381 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:05.421180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.421111 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:52:05.422197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.422116 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:05.452326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.452274 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:05.517318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.452640 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:52:05.517318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.451215 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:05.517318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.452221 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:05.546070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.538396 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:05.814099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.734094 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:05.814099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.796817 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:05.814099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:05.796890 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:06.050070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.036157 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:06.118457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.118333 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded" Oct 30 05:52:06.252069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.240635 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:52:06.294068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.291228 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:52:06.306068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.302093 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:52:06.407908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.407820 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:52:06.423196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.423168 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:52:06.437239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.437206 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:52:06.468218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.468170 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:52:06.752539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.752509 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:52:06.752739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.752714 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:06.752823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:06.752765 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:07.753729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:07.753695 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" start-of-body= Oct 30 05:52:07.754170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:07.753755 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.149:15021/healthz/ready\": dial tcp 10.133.0.149:15021: connect: connection refused" Oct 30 05:52:09.078302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:09.078272 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:52:09.079425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:09.079403 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:52:10.088067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:10.084150 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8"] Oct 30 05:52:10.116969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:10.116933 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-968d7f55-8pws8"] Oct 30 05:52:11.674822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:11.674777 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d3e4d9-7217-448d-9542-c41032805a57" path="/var/lib/kubelet/pods/78d3e4d9-7217-448d-9542-c41032805a57/volumes" Oct 30 05:52:12.037288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:12.037261 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw"] Oct 30 05:52:12.048148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:12.048111 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk"] Oct 30 05:52:12.053576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:12.053559 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa02d109-gjphw"] Oct 30 05:52:12.058882 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:12.058863 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1c771a3e-qx6bk"] Oct 30 05:52:13.673717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:13.673683 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3785394b-1444-45c6-a068-04ceebee95c9" path="/var/lib/kubelet/pods/3785394b-1444-45c6-a068-04ceebee95c9/volumes" Oct 30 05:52:13.674196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:13.674148 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0" path="/var/lib/kubelet/pods/7941aa2f-2fcc-49ac-95b2-3163c6c0f0f0/volumes" Oct 30 05:52:15.709282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.709244 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 05:52:15.710217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.710195 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f5a57c41-036a-4f5c-a7db-6f0ded6af417" containerName="revoke-keys" Oct 30 05:52:15.710217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.710216 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5a57c41-036a-4f5c-a7db-6f0ded6af417" containerName="revoke-keys" Oct 30 05:52:15.710502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.710487 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5a57c41-036a-4f5c-a7db-6f0ded6af417" containerName="revoke-keys" Oct 30 05:52:15.715970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.715950 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.731710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.731690 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6b5b1e\"" Oct 30 05:52:15.735919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.735898 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 05:52:15.773499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.773474 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 05:52:15.780154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.780132 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:15.800826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.800799 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 05:52:15.820386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.820339 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6b5b1e-dockercfg-cqxsr\"" Oct 30 05:52:15.824157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.824120 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqrg\" (UniqueName: \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:15.824668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.824602 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.824971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.824952 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:15.825323 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.825296 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgwzp\" (UniqueName: \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.937067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.932625 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqrg\" (UniqueName: \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:15.937067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.932797 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.937067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.932928 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:15.937067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.933099 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgwzp\" (UniqueName: \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.937067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.934722 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:15.959144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:15.959106 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:16.219887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.219862 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgwzp\" (UniqueName: \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\") pod \"payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:16.229269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.229147 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqrg\" (UniqueName: \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\") pod \"payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:16.343689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.341650 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:16.401710 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.401671 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:16.784848 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:16.784810 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12401bb7_cf57_45fd_b716_0f8b020e48ba.slice/crio-3dee8de708f084a045570d2dc191fb6c91544513e01052f97baf7aede3123e02 WatchSource:0}: Error finding container 3dee8de708f084a045570d2dc191fb6c91544513e01052f97baf7aede3123e02: Status 404 returned error can't find the container with id 3dee8de708f084a045570d2dc191fb6c91544513e01052f97baf7aede3123e02 Oct 30 05:52:16.788680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.788653 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 05:52:16.808731 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:16.808702 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f943a43_895b_4347_ae66_93b478d336e2.slice/crio-2d38c34022875f9b03b14346ee98abc1bc490ec3eb65ac7b50600874aabf4146 WatchSource:0}: Error finding container 2d38c34022875f9b03b14346ee98abc1bc490ec3eb65ac7b50600874aabf4146: Status 404 returned error can't find the container with id 2d38c34022875f9b03b14346ee98abc1bc490ec3eb65ac7b50600874aabf4146 Oct 30 05:52:16.825491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.825465 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 05:52:16.938588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.938562 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq"] Oct 30 05:52:16.943484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.943466 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:16.959322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.959300 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:52:16.962700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.962677 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq"] Oct 30 05:52:16.966809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.966786 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:52:16.980473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.980439 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:52:16.980614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:16.980482 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:52:17.036801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.036727 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.037002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.036939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zp5r\" (UniqueName: \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.037002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.036977 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.137938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.137906 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.138146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.138109 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zp5r\" (UniqueName: \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.138212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.138146 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.138723 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.138704 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.140330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.140312 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.161568 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.161549 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zp5r\" (UniqueName: \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\") pod \"maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.259616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.259585 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:17.554016 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.553986 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" event={"ID":"12401bb7-cf57-45fd-b716-0f8b020e48ba","Type":"ContainerStarted","Data":"076af070dea7b7308f62f09d9e3073313d82989c5fa521f236fc3f409dcf7e67"} Oct 30 05:52:17.554226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.554036 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" event={"ID":"12401bb7-cf57-45fd-b716-0f8b020e48ba","Type":"ContainerStarted","Data":"3dee8de708f084a045570d2dc191fb6c91544513e01052f97baf7aede3123e02"} Oct 30 05:52:17.554226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.554079 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:17.555846 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.555817 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" event={"ID":"4f943a43-895b-4347-ae66-93b478d336e2","Type":"ContainerStarted","Data":"19d6310029f68204e47c79901b5b69257fee36b768594346e685cbfe5e1292ec"} Oct 30 05:52:17.555974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.555862 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" event={"ID":"4f943a43-895b-4347-ae66-93b478d336e2","Type":"ContainerStarted","Data":"2d38c34022875f9b03b14346ee98abc1bc490ec3eb65ac7b50600874aabf4146"} Oct 30 05:52:17.555974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.555883 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:17.574216 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:17.574190 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb3d880_087f_4655_be3c_42f318de3d3f.slice/crio-4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4 WatchSource:0}: Error finding container 4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4: Status 404 returned error can't find the container with id 4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4 Oct 30 05:52:17.577995 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.577971 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq"] Oct 30 05:52:17.604450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.604390 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" podStartSLOduration=2.6043735679999998 podStartE2EDuration="2.604373568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:17.602619253 +0000 UTC m=+1196.042996446" watchObservedRunningTime="2026-10-30 05:52:17.604373568 +0000 UTC m=+1196.044750760" Oct 30 05:52:17.655970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:17.655927 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" podStartSLOduration=2.655913731 podStartE2EDuration="2.655913731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:17.653648858 +0000 UTC m=+1196.094026049" watchObservedRunningTime="2026-10-30 05:52:17.655913731 +0000 UTC m=+1196.096290920" Oct 30 05:52:18.567419 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:18.567379 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" event={"ID":"8fb3d880-087f-4655-be3c-42f318de3d3f","Type":"ContainerStarted","Data":"b19fb05cb4cace52c3656e6b23614ba05f2063f96ab8a1b832453a51a18f0a35"} Oct 30 05:52:18.567876 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:18.567480 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" event={"ID":"8fb3d880-087f-4655-be3c-42f318de3d3f","Type":"ContainerStarted","Data":"4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4"} Oct 30 05:52:18.614689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:18.614641 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" podStartSLOduration=2.6146294340000003 podStartE2EDuration="2.614629434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:18.610332679 +0000 UTC m=+1197.050709872" watchObservedRunningTime="2026-10-30 05:52:18.614629434 +0000 UTC m=+1197.055006625" Oct 30 05:52:19.643091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:19.643023 6171 generic.go:356] "Generic (PLEG): container finished" podID="8fb3d880-087f-4655-be3c-42f318de3d3f" containerID="b19fb05cb4cace52c3656e6b23614ba05f2063f96ab8a1b832453a51a18f0a35" exitCode=0 Oct 30 05:52:19.643830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:19.643805 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" event={"ID":"8fb3d880-087f-4655-be3c-42f318de3d3f","Type":"ContainerDied","Data":"b19fb05cb4cace52c3656e6b23614ba05f2063f96ab8a1b832453a51a18f0a35"} Oct 30 05:52:20.815913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.815888 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:20.858594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.858565 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\" (UniqueName: \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\") pod \"8fb3d880-087f-4655-be3c-42f318de3d3f\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " Oct 30 05:52:20.858731 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.858685 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\") pod \"8fb3d880-087f-4655-be3c-42f318de3d3f\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " Oct 30 05:52:20.858794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.858744 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\") pod \"8fb3d880-087f-4655-be3c-42f318de3d3f\" (UID: \"8fb3d880-087f-4655-be3c-42f318de3d3f\") " Oct 30 05:52:20.860211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.860186 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca" pod "8fb3d880-087f-4655-be3c-42f318de3d3f" (UID: "8fb3d880-087f-4655-be3c-42f318de3d3f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:20.861802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.861771 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca" pod "8fb3d880-087f-4655-be3c-42f318de3d3f" (UID: "8fb3d880-087f-4655-be3c-42f318de3d3f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:20.862302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.862278 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r" pod "8fb3d880-087f-4655-be3c-42f318de3d3f" (UID: "8fb3d880-087f-4655-be3c-42f318de3d3f"). InnerVolumeSpecName "kube-api-access-4zp5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:20.961470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.961397 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zp5r\" (UniqueName: \"kubernetes.io/projected/8fb3d880-087f-4655-be3c-42f318de3d3f-kube-api-access-4zp5r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:20.961470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.961424 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:20.961470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:20.961434 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fb3d880-087f-4655-be3c-42f318de3d3f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:21.682396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:21.682367 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" Oct 30 05:52:21.682585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:21.682365 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq" event={"ID":"8fb3d880-087f-4655-be3c-42f318de3d3f","Type":"ContainerDied","Data":"4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4"} Oct 30 05:52:21.682585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:21.682508 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4818a11b8d15f5f3556e40941604d59e25efa2de6488f51148d2a489da872ba4" Oct 30 05:52:23.046126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.046093 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2"] Oct 30 05:52:23.057229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.057204 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-449bb3e2-vffs2"] Oct 30 05:52:23.674748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.674718 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e29ce9-d30b-4e39-b0d6-a62293f929c8" path="/var/lib/kubelet/pods/d6e29ce9-d30b-4e39-b0d6-a62293f929c8/volumes" Oct 30 05:52:23.974463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.974439 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl"] Oct 30 05:52:23.975212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.975198 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8fb3d880-087f-4655-be3c-42f318de3d3f" containerName="revoke-keys" Oct 30 05:52:23.975262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.975214 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb3d880-087f-4655-be3c-42f318de3d3f" containerName="revoke-keys" Oct 30 05:52:23.975383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.975374 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb3d880-087f-4655-be3c-42f318de3d3f" containerName="revoke-keys" Oct 30 05:52:23.981284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:23.981268 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.002183 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.002154 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl"] Oct 30 05:52:24.002856 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.002834 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:52:24.002975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.002930 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:52:24.002975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.002962 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:52:24.003227 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.003209 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:52:24.026504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.026471 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.026749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.026711 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.027275 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.026749 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdmbk\" (UniqueName: \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.044370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.044348 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9"] Oct 30 05:52:24.050866 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.050839 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e055c622-82sp9"] Oct 30 05:52:24.131064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.131006 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.131223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.131209 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.131298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.131247 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdmbk\" (UniqueName: \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.131669 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.131650 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.133721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.133700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.154713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.154694 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdmbk\" (UniqueName: \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\") pod \"maas-api-revoke-keys-e2e-ait-709c4831-4qwgl\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.298500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.298413 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:24.696120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.683815 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:52:24.799077 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:24.792532 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c29b4c_42cf_4e9e_bcfa_3dab19237c2b.slice/crio-99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d WatchSource:0}: Error finding container 99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d: Status 404 returned error can't find the container with id 99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d Oct 30 05:52:24.836223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:24.835343 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl"] Oct 30 05:52:25.837066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:25.830297 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8eb26885-8ce8-4af2-8736-606701a15d7e" path="/var/lib/kubelet/pods/8eb26885-8ce8-4af2-8736-606701a15d7e/volumes" Oct 30 05:52:25.979086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:25.961633 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" event={"ID":"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b","Type":"ContainerStarted","Data":"776f33a5812825473400cdab9ce728154731987df55934b6ba9a8a93f91592db"} Oct 30 05:52:25.979086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:25.961701 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" event={"ID":"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b","Type":"ContainerStarted","Data":"99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d"} Oct 30 05:52:26.096083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:26.088827 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" podStartSLOduration=3.088805088 podStartE2EDuration="3.088805088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:26.053952086 +0000 UTC m=+1204.494329278" watchObservedRunningTime="2026-10-30 05:52:26.088805088 +0000 UTC m=+1204.529182280" Oct 30 05:52:26.384125 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:52:26.381232 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:52:27.108066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.106099 6171 generic.go:356] "Generic (PLEG): container finished" podID="f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" containerID="776f33a5812825473400cdab9ce728154731987df55934b6ba9a8a93f91592db" exitCode=0 Oct 30 05:52:27.108066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.107619 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" event={"ID":"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b","Type":"ContainerDied","Data":"776f33a5812825473400cdab9ce728154731987df55934b6ba9a8a93f91592db"} Oct 30 05:52:27.154073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.153102 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5"] Oct 30 05:52:27.426377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.426276 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7"] Oct 30 05:52:27.429128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.426694 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:27.501823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.501769 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:27.703904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.701664 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:27.923764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.923732 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c28447a3-8skr5"] Oct 30 05:52:27.943070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.938212 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:52:27.965124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:27.954715 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e81cc108-7wkd7"] Oct 30 05:52:28.028071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.027126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:28.028071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.027543 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.431686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.431634 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.432625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.432251 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.434173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.433337 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:52:28.446342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.446298 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.460473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.460439 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:52:28.614071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.600189 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.694605 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.694296 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:52:28.733070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.703171 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.733070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.703548 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:52:28.933957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.924308 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 05:52:28.949145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.941756 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:28.949145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.942362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:52:28.972433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:28.971268 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 05:52:29.037275 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.037217 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:29.064099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.041088 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:52:29.064099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.042097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:52:29.064099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.042158 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:52:29.501072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.487137 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:29.501072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.495193 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:52:29.507068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.503140 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:29.519125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.516114 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 05:52:29.574068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.536602 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:52:29.574068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.562220 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:29.624194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.618010 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:29.654080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.634079 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:52:29.923120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.875632 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 05:52:29.971306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:29.943586 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:30.047100 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.042586 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:52:30.071073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.051421 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 05:52:30.071073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.051530 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:52:30.114776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.114736 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5751de18-3728-4198-a097-d149fbddbc61" path="/var/lib/kubelet/pods/5751de18-3728-4198-a097-d149fbddbc61/volumes" Oct 30 05:52:30.140070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.115782 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcf3fb25-226b-41ab-9b75-b0bf81a864f1" path="/var/lib/kubelet/pods/dcf3fb25-226b-41ab-9b75-b0bf81a864f1/volumes" Oct 30 05:52:30.496957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.496909 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:30.538260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.538206 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:30.566083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.562485 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:30.626064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.626002 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:30.730765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.699721 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:52:30.730765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.723133 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:52:30.800161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.786015 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:30.964070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.963543 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:30.979077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:30.965743 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:31.015332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.015286 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\" (UniqueName: \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\") pod \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " Oct 30 05:52:31.027079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.023632 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\") pod \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " Oct 30 05:52:31.107067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.047803 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:52:31.107067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.061502 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:31.129069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.062328 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca" pod "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" (UID: "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:31.129312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.123928 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\") pod \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\" (UID: \"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b\") " Oct 30 05:52:31.135074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.130432 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:31.174125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.151602 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca" pod "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" (UID: "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:31.222069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.205086 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk" pod "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" (UID: "f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b"). InnerVolumeSpecName "kube-api-access-fdmbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:31.304078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.283069 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdmbk\" (UniqueName: \"kubernetes.io/projected/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-kube-api-access-fdmbk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:31.304078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.283106 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:31.660307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.660085 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:31.720070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.703485 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:52:31.983334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:31.983072 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:32.021078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.002102 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:32.151070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.125942 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:32.161987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.128219 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": context deadline exceeded" Oct 30 05:52:32.250223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.250121 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:52:32.513134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.512101 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:52:32.513134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.512195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:52:32.513134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.512254 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:52:32.805128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.791239 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:32.805128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.792714 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:32.805128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.792936 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:52:32.805128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.793128 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 05:52:32.975916 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:32.975878 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-cdqfb" podUID="9e53b670-2539-4a61-a1aa-fe5e43005508" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.133.0.16:6080/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.160134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.158456 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.292108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.272742 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.448973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.442630 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8"] Oct 30 05:52:33.455812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.455770 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" event={"ID":"f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b","Type":"ContainerDied","Data":"99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d"} Oct 30 05:52:33.456207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.456184 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99b4592b3830e64f4d486ffd907237700322a45ab19fc334431ac78024c8769d" Oct 30 05:52:33.456358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.456126 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl" Oct 30 05:52:33.472276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.472237 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" podUID="405c2936-8199-42f0-b09f-3878abb8a351" containerName="payload-processing" containerID="cri-o://ef2b4fe2238d621f216e747caf9bcae3344159f47cd5373cf8b679868bf6eb51" gracePeriod=25 Oct 30 05:52:33.588107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.583167 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.588107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.584554 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 05:52:33.646461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.646431 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:52:33.814167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.726588 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" podUID="d2777af9-55f2-492c-ae04-615c9827aa22" containerName="payload-pre-processing" containerID="cri-o://1767376e0e9941808fc6ba2180aa4bc9a346e2ad6952af312d98b776365a0d58" gracePeriod=25 Oct 30 05:52:33.814167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.808973 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.814167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.809262 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:52:33.895139 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:33.840306 6171 logging.go:55] [core] [Channel #1757 SubChannel #1758]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.146:50052", ServerName: "10.133.0.146:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.146:50052: connect: connection refused" Oct 30 05:52:33.895139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.841155 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:52:33.963689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.963198 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:33.994683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:33.994650 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:52:34.262883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.262857 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:52:34.263157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.263145 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:52:34.263272 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.263262 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c8328ad-4sdr8"] Oct 30 05:52:34.263411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.263402 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:52:34.263512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.263504 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:52:34.265473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.265362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:52:34.479011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.478976 6171 generic.go:356] "Generic (PLEG): container finished" podID="405c2936-8199-42f0-b09f-3878abb8a351" containerID="ef2b4fe2238d621f216e747caf9bcae3344159f47cd5373cf8b679868bf6eb51" exitCode=0 Oct 30 05:52:34.479469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.479445 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" event={"ID":"405c2936-8199-42f0-b09f-3878abb8a351","Type":"ContainerDied","Data":"ef2b4fe2238d621f216e747caf9bcae3344159f47cd5373cf8b679868bf6eb51"} Oct 30 05:52:34.481869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.481838 6171 generic.go:356] "Generic (PLEG): container finished" podID="d2777af9-55f2-492c-ae04-615c9827aa22" containerID="1767376e0e9941808fc6ba2180aa4bc9a346e2ad6952af312d98b776365a0d58" exitCode=0 Oct 30 05:52:34.482878 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.482854 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" event={"ID":"d2777af9-55f2-492c-ae04-615c9827aa22","Type":"ContainerDied","Data":"1767376e0e9941808fc6ba2180aa4bc9a346e2ad6952af312d98b776365a0d58"} Oct 30 05:52:34.625483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.625457 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:52:34.677511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.677485 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\" (UniqueName: \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\") pod \"405c2936-8199-42f0-b09f-3878abb8a351\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " Oct 30 05:52:34.677848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.677826 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\") pod \"405c2936-8199-42f0-b09f-3878abb8a351\" (UID: \"405c2936-8199-42f0-b09f-3878abb8a351\") " Oct 30 05:52:34.678229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.678205 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume" pod "405c2936-8199-42f0-b09f-3878abb8a351" (UID: "405c2936-8199-42f0-b09f-3878abb8a351"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:34.679191 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.679162 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/405c2936-8199-42f0-b09f-3878abb8a351-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:34.729114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.720511 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt" pod "405c2936-8199-42f0-b09f-3878abb8a351" (UID: "405c2936-8199-42f0-b09f-3878abb8a351"). InnerVolumeSpecName "kube-api-access-hpfjt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:34.751538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.751515 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:52:34.773355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.773321 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" podUID="405c2936-8199-42f0-b09f-3878abb8a351" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.146:50052\" within 1s: context deadline exceeded" Oct 30 05:52:34.794213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.780380 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\" (UniqueName: \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\") pod \"d2777af9-55f2-492c-ae04-615c9827aa22\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " Oct 30 05:52:34.817198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.816347 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\") pod \"d2777af9-55f2-492c-ae04-615c9827aa22\" (UID: \"d2777af9-55f2-492c-ae04-615c9827aa22\") " Oct 30 05:52:34.817629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.817372 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfjt\" (UniqueName: \"kubernetes.io/projected/405c2936-8199-42f0-b09f-3878abb8a351-kube-api-access-hpfjt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:34.818527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.817754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume" pod "d2777af9-55f2-492c-ae04-615c9827aa22" (UID: "d2777af9-55f2-492c-ae04-615c9827aa22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:34.820244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.819192 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99" pod "d2777af9-55f2-492c-ae04-615c9827aa22" (UID: "d2777af9-55f2-492c-ae04-615c9827aa22"). InnerVolumeSpecName "kube-api-access-k6q99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:34.955741 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.955234 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6q99\" (UniqueName: \"kubernetes.io/projected/d2777af9-55f2-492c-ae04-615c9827aa22-kube-api-access-k6q99\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:34.955741 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:34.955278 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d2777af9-55f2-492c-ae04-615c9827aa22-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:35.128455 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.128363 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x"] Oct 30 05:52:35.135197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.135169 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d8586d8-fc75x"] Oct 30 05:52:35.550163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.538941 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" event={"ID":"d2777af9-55f2-492c-ae04-615c9827aa22","Type":"ContainerDied","Data":"0d88fadccf39c31268a620de92a2de2fe6a10c4c0fa297b1f92c211ea52da3fb"} Oct 30 05:52:35.550163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.539023 6171 scope.go:122] "RemoveContainer" containerID="1767376e0e9941808fc6ba2180aa4bc9a346e2ad6952af312d98b776365a0d58" Oct 30 05:52:35.550163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.539338 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47" Oct 30 05:52:35.563301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.562548 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" event={"ID":"405c2936-8199-42f0-b09f-3878abb8a351","Type":"ContainerDied","Data":"8cd865f5b90bdbb74e37ef442bd50ab2a30378fb9e6e3c160a7be16a4b44d2da"} Oct 30 05:52:35.563301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.562671 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m" Oct 30 05:52:35.789091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.787524 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13296f31-2f34-4cb2-83f6-f6bf3c54bf4d" path="/var/lib/kubelet/pods/13296f31-2f34-4cb2-83f6-f6bf3c54bf4d/volumes" Oct 30 05:52:35.789091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.788360 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efd1f3e6-7a65-4c97-a924-fc296c385ef5" path="/var/lib/kubelet/pods/efd1f3e6-7a65-4c97-a924-fc296c385ef5/volumes" Oct 30 05:52:35.830334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.828872 6171 scope.go:122] "RemoveContainer" containerID="ef2b4fe2238d621f216e747caf9bcae3344159f47cd5373cf8b679868bf6eb51" Oct 30 05:52:35.852528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.852443 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:52:35.888480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.888408 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-54d4943b-8587c75899-pwc9m"] Oct 30 05:52:35.901836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.901777 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:52:35.921036 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:35.920999 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-54d4943b-77458c6b99-gpb47"] Oct 30 05:52:36.104118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:36.092588 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh"] Oct 30 05:52:36.271822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:36.267929 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ec72c1-57zxh"] Oct 30 05:52:37.297851 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.297816 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq"] Oct 30 05:52:37.308023 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.307996 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-54d4943b-kkzrq"] Oct 30 05:52:37.479292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.473409 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:52:37.479292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.474250 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" podUID="927271a8-8f85-454c-a5a4-56206980d6c4" containerName="istio-proxy" containerID="cri-o://be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5" gracePeriod=30 Oct 30 05:52:37.797121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.791411 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405c2936-8199-42f0-b09f-3878abb8a351" path="/var/lib/kubelet/pods/405c2936-8199-42f0-b09f-3878abb8a351/volumes" Oct 30 05:52:37.849083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.804556 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5016c6bb-9b58-45e1-885f-cee9ddf915d7" path="/var/lib/kubelet/pods/5016c6bb-9b58-45e1-885f-cee9ddf915d7/volumes" Oct 30 05:52:37.849083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.805374 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb3d880-087f-4655-be3c-42f318de3d3f" path="/var/lib/kubelet/pods/8fb3d880-087f-4655-be3c-42f318de3d3f/volumes" Oct 30 05:52:37.849083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:37.806012 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2777af9-55f2-492c-ae04-615c9827aa22" path="/var/lib/kubelet/pods/d2777af9-55f2-492c-ae04-615c9827aa22/volumes" Oct 30 05:52:38.702344 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.702303 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:52:38.710853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710820 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d2777af9-55f2-492c-ae04-615c9827aa22" containerName="payload-pre-processing" Oct 30 05:52:38.710853 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710853 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2777af9-55f2-492c-ae04-615c9827aa22" containerName="payload-pre-processing" Oct 30 05:52:38.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710936 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" containerName="revoke-keys" Oct 30 05:52:38.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710947 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" containerName="revoke-keys" Oct 30 05:52:38.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710973 6171 cpu_manager.go:414] "containerMap: removing container" podUID="405c2936-8199-42f0-b09f-3878abb8a351" containerName="payload-processing" Oct 30 05:52:38.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.710982 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405c2936-8199-42f0-b09f-3878abb8a351" containerName="payload-processing" Oct 30 05:52:38.711479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.711457 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" containerName="revoke-keys" Oct 30 05:52:38.711623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.711488 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="405c2936-8199-42f0-b09f-3878abb8a351" containerName="payload-processing" Oct 30 05:52:38.711623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.711516 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2777af9-55f2-492c-ae04-615c9827aa22" containerName="payload-pre-processing" Oct 30 05:52:38.719743 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.719717 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:52:38.720023 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.719809 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.743083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.743023 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a095fc35-openshift-default-dockercfg-w4zms\"" Oct 30 05:52:38.810293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810261 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810307 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810473 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810555 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810666 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810866 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810850 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.810907 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810889 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.811001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.810983 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.811171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.811155 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.945775 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.945870 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946022 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946080 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946252 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946338 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946368 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.946470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.946432 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.954114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.947171 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.954114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.949895 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.954114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.950582 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.954114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.951101 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.954114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.953520 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.990818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.989689 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:38.992354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:38.992321 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:39.018159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.017708 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:39.022401 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.022367 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\") pod \"e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:39.044690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.044374 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:39.934430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.934397 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd"] Oct 30 05:52:39.934983 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.934963 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d1e101f-dsvhd"] Oct 30 05:52:39.935225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.935208 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl"] Oct 30 05:52:39.974167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:39.971613 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-709c4831-4qwgl"] Oct 30 05:52:40.626352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:40.624362 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:40.626352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:40.624440 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:40.626352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:40.624472 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:52:40.627014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:40.626994 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:52:41.048955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.048921 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t"] Oct 30 05:52:41.059486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.059453 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a49ea64-cmt6t"] Oct 30 05:52:41.119208 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.119167 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerStarted","Data":"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593"} Oct 30 05:52:41.119379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.119276 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerStarted","Data":"9542d3f4595b0604c8e8fbb1b404923bcd1320a6a07e46bb2b8e77fb5c6014c3"} Oct 30 05:52:41.172502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.172448 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podStartSLOduration=3.172433719 podStartE2EDuration="3.172433719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:41.167448692 +0000 UTC m=+1219.607825888" watchObservedRunningTime="2026-10-30 05:52:41.172433719 +0000 UTC m=+1219.612810910" Oct 30 05:52:41.889075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.878965 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e5ab780-b600-45e4-8806-d954c4f17e6e" path="/var/lib/kubelet/pods/1e5ab780-b600-45e4-8806-d954c4f17e6e/volumes" Oct 30 05:52:41.889075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.879773 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5253fdb2-30fb-4116-b9e8-20220b6a06cf" path="/var/lib/kubelet/pods/5253fdb2-30fb-4116-b9e8-20220b6a06cf/volumes" Oct 30 05:52:41.889075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:41.885102 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b" path="/var/lib/kubelet/pods/f8c29b4c-42cf-4e9e-bcfa-3dab19237c2b/volumes" Oct 30 05:52:42.045506 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:42.045467 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:42.051307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:42.051284 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:42.189194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:42.189114 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:42.203404 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:42.203360 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:52:43.141672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.141629 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg"] Oct 30 05:52:43.147780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.147755 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7e81b3e0-m65cg"] Oct 30 05:52:43.735395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.735359 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7086963e-3b9c-4cf5-a783-6a3f30e306e2" path="/var/lib/kubelet/pods/7086963e-3b9c-4cf5-a783-6a3f30e306e2/volumes" Oct 30 05:52:43.739859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.738367 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:52:43.814924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.814887 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.814948 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815113 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815234 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815229 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:43.815356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815302 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815329 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:43.815356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815341 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815370 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815424 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\") pod \"927271a8-8f85-454c-a5a4-56206980d6c4\" (UID: \"927271a8-8f85-454c-a5a4-56206980d6c4\") " Oct 30 05:52:43.815786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.815714 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816126 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816333 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816717 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816739 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816752 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/927271a8-8f85-454c-a5a4-56206980d6c4-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816765 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.816806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.816780 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.818325 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.818303 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:43.819425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.819386 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:43.819425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.819420 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:43.820611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.820590 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz" pod "927271a8-8f85-454c-a5a4-56206980d6c4" (UID: "927271a8-8f85-454c-a5a4-56206980d6c4"). InnerVolumeSpecName "kube-api-access-5mdkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:43.933131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.928656 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/927271a8-8f85-454c-a5a4-56206980d6c4-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.933131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.928696 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/927271a8-8f85-454c-a5a4-56206980d6c4-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.933131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.928733 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mdkz\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-kube-api-access-5mdkz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:43.933131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:43.928754 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/927271a8-8f85-454c-a5a4-56206980d6c4-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:52:44.211540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.211438 6171 generic.go:356] "Generic (PLEG): container finished" podID="927271a8-8f85-454c-a5a4-56206980d6c4" containerID="be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5" exitCode=0 Oct 30 05:52:44.211540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.211513 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" Oct 30 05:52:44.211540 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.211523 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" event={"ID":"927271a8-8f85-454c-a5a4-56206980d6c4","Type":"ContainerDied","Data":"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5"} Oct 30 05:52:44.212157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.211623 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx" event={"ID":"927271a8-8f85-454c-a5a4-56206980d6c4","Type":"ContainerDied","Data":"0d795481aa2a5144404b9309b2b6eb404c87c4c202cfddf317d6892dc67b033e"} Oct 30 05:52:44.212157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.211654 6171 scope.go:122] "RemoveContainer" containerID="be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5" Oct 30 05:52:44.226879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.226854 6171 scope.go:122] "RemoveContainer" containerID="be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5" Oct 30 05:52:44.227207 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:52:44.227179 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5\": container with ID starting with be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5 not found: ID does not exist" containerID="be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5" Oct 30 05:52:44.227342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.227216 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5"} err="failed to get container status \"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5\": rpc error: code = NotFound desc = could not find container \"be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5\": container with ID starting with be06230bd276f7940dcb73e9c740ad8f3f67171523213eea3ab9c7e85ed0e3d5 not found: ID does not exist" Oct 30 05:52:44.269916 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.269890 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:52:44.284237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.284183 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-54d4943b-gw-openshift-default-574d4fd98f-95ttx"] Oct 30 05:52:44.967370 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.967343 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:52:44.968226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.968203 6171 cpu_manager.go:414] "containerMap: removing container" podUID="927271a8-8f85-454c-a5a4-56206980d6c4" containerName="istio-proxy" Oct 30 05:52:44.968226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.968223 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="927271a8-8f85-454c-a5a4-56206980d6c4" containerName="istio-proxy" Oct 30 05:52:44.968421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.968411 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="927271a8-8f85-454c-a5a4-56206980d6c4" containerName="istio-proxy" Oct 30 05:52:44.974329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.974306 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:44.990322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:44.990297 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-a095fc35\"" Oct 30 05:52:45.172075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.148344 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-a095fc35-dockercfg-mzsvx\"" Oct 30 05:52:45.245069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.232307 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:52:45.508838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.508147 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:52:45.514222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.514193 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.518643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.518617 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:52:45.547427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.547401 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqsjz\" (UniqueName: \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.547566 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.547551 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.649213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.649180 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfj2t\" (UniqueName: \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.649376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.649296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqsjz\" (UniqueName: \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.649376 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.649352 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.649448 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.649386 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.650003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.649985 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.675574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.675544 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="927271a8-8f85-454c-a5a4-56206980d6c4" path="/var/lib/kubelet/pods/927271a8-8f85-454c-a5a4-56206980d6c4/volumes" Oct 30 05:52:45.680433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.680409 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqsjz\" (UniqueName: \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\") pod \"payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:45.751361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.751321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfj2t\" (UniqueName: \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.751543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.751426 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.752092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.752035 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.779322 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.779264 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfj2t\" (UniqueName: \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\") pod \"payload-processing-e2e-ait-a095fc35-54d6b699-9brgh\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.830765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.830732 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:45.906171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:45.905122 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:46.151399 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:46.151370 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf2dd47_05a9_41a7_8832_02a30fe858b4.slice/crio-034104d600abcec8721442ee20457dee70a29a353dbe9c97e0747c04fba12ade WatchSource:0}: Error finding container 034104d600abcec8721442ee20457dee70a29a353dbe9c97e0747c04fba12ade: Status 404 returned error can't find the container with id 034104d600abcec8721442ee20457dee70a29a353dbe9c97e0747c04fba12ade Oct 30 05:52:46.155022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.155001 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:52:46.233754 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:52:46.233730 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b30fa8_581a_4a98_ad52_e6e5a2170b82.slice/crio-51e27d1d7b1fca089929e5b2bc5886cb8486fe9d92a22b2f86ccd046d9cf43bd WatchSource:0}: Error finding container 51e27d1d7b1fca089929e5b2bc5886cb8486fe9d92a22b2f86ccd046d9cf43bd: Status 404 returned error can't find the container with id 51e27d1d7b1fca089929e5b2bc5886cb8486fe9d92a22b2f86ccd046d9cf43bd Oct 30 05:52:46.239383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.235723 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:52:46.582193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.582151 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" event={"ID":"caf2dd47-05a9-41a7-8832-02a30fe858b4","Type":"ContainerStarted","Data":"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183"} Oct 30 05:52:46.582662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.582241 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" event={"ID":"caf2dd47-05a9-41a7-8832-02a30fe858b4","Type":"ContainerStarted","Data":"034104d600abcec8721442ee20457dee70a29a353dbe9c97e0747c04fba12ade"} Oct 30 05:52:46.582662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.582308 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:46.584200 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.584176 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" event={"ID":"a1b30fa8-581a-4a98-ad52-e6e5a2170b82","Type":"ContainerStarted","Data":"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee"} Oct 30 05:52:46.584380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.584364 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" event={"ID":"a1b30fa8-581a-4a98-ad52-e6e5a2170b82","Type":"ContainerStarted","Data":"51e27d1d7b1fca089929e5b2bc5886cb8486fe9d92a22b2f86ccd046d9cf43bd"} Oct 30 05:52:46.584570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.584559 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:52:46.628129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.628082 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" podStartSLOduration=1.628066924 podStartE2EDuration="1.628066924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:46.622387182 +0000 UTC m=+1225.062764374" watchObservedRunningTime="2026-10-30 05:52:46.628066924 +0000 UTC m=+1225.068444115" Oct 30 05:52:46.665271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:46.665223 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" podStartSLOduration=2.665209744 podStartE2EDuration="2.665209744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:46.663349388 +0000 UTC m=+1225.103726579" watchObservedRunningTime="2026-10-30 05:52:46.665209744 +0000 UTC m=+1225.105586963" Oct 30 05:52:52.117073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:52.112108 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7"] Oct 30 05:52:52.198062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:52.183382 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3021f688-c24k7"] Oct 30 05:52:53.673242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:53.673210 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43634ed9-047e-4f71-abad-d76717c3a8e9" path="/var/lib/kubelet/pods/43634ed9-047e-4f71-abad-d76717c3a8e9/volumes" Oct 30 05:52:56.381413 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:52:56.381379 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:52:57.635033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:57.635000 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:52:57.635545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:52:57.635128 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:53:00.726956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.726917 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 05:53:00.734590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.734566 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.765939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.765908 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 05:53:00.888789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.887304 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c73fcb-openshift-default-dockercfg-rtknk\"" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907144 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907280 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94p8\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907397 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907437 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907479 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907557 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907712 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907760 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:00.912288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:00.907873 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.038119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.012003 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.145110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.126578 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94p8\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.145538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.145500 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.145784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.145765 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.145942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.145928 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.146221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.146120 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.146528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.146505 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.146728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.146705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.151063 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.146941 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.162085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.157705 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.162085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.134550 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.162085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.158018 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.186065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.180083 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.186065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.181151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.239108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.216898 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94p8\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.242974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.242929 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.266346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.266276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.287857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.287817 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.356339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.356250 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:01.742513 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:01.742484 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7756216c_0f8a_4f1b_bc98_b31c0c561ac1.slice/crio-ad080e3022222977d2bd593e19dde34c5101b591586c1896cd78776ad6b83c79 WatchSource:0}: Error finding container ad080e3022222977d2bd593e19dde34c5101b591586c1896cd78776ad6b83c79: Status 404 returned error can't find the container with id ad080e3022222977d2bd593e19dde34c5101b591586c1896cd78776ad6b83c79 Oct 30 05:53:01.745252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.745213 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:01.745378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.745284 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:01.745378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.745314 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:01.746379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:01.746358 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 05:53:02.335372 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:02.335333 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerStarted","Data":"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7"} Oct 30 05:53:02.335571 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:02.335391 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerStarted","Data":"ad080e3022222977d2bd593e19dde34c5101b591586c1896cd78776ad6b83c79"} Oct 30 05:53:02.357830 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:02.357804 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:02.363233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:02.363204 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:02.426279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:02.426213 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podStartSLOduration=2.426193207 podStartE2EDuration="2.426193207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:02.40595093 +0000 UTC m=+1240.846328135" watchObservedRunningTime="2026-10-30 05:53:02.426193207 +0000 UTC m=+1240.866570400" Oct 30 05:53:03.381072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:03.369340 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:03.396070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:03.395292 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:53:07.022757 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.022721 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 05:53:07.027884 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.027866 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.045604 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.045530 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-c73fcb\"" Oct 30 05:53:07.057014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.056986 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-c73fcb-dockercfg-l5fwz\"" Oct 30 05:53:07.065861 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.065833 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 05:53:07.235675 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.197999 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcmvd\" (UniqueName: \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.235675 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.198173 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.396299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.391198 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcmvd\" (UniqueName: \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.396299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.391359 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.396299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.392784 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.464312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.462943 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 05:53:07.547371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.496824 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.547371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.500887 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcmvd\" (UniqueName: \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\") pod \"payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.547371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.504089 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 05:53:07.598518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.598484 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmpct\" (UniqueName: \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.598721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.598680 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.665333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.665252 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:07.700199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.700166 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmpct\" (UniqueName: \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.700379 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.700361 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.701166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.701141 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.771314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.771278 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmpct\" (UniqueName: \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\") pod \"payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:07.861340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:07.861301 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:08.104933 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.101907 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 05:53:08.282948 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:08.282915 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod209835cf_0fac_42a7_a57f_a13c76ddcff0.slice/crio-7b4f01f3536cc0949a73f033a7e25395306efd361410bc09891bf2ace9031f4d WatchSource:0}: Error finding container 7b4f01f3536cc0949a73f033a7e25395306efd361410bc09891bf2ace9031f4d: Status 404 returned error can't find the container with id 7b4f01f3536cc0949a73f033a7e25395306efd361410bc09891bf2ace9031f4d Oct 30 05:53:08.285696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.285666 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 05:53:08.511544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.511508 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" event={"ID":"9e20720e-468f-4180-9e47-9dce2ec0f1c8","Type":"ContainerStarted","Data":"c4c5255b38c5d10b2ed586267f5da2d1c83ffb33cf9e90d3f954773c37b49f74"} Oct 30 05:53:08.511701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.511563 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" event={"ID":"9e20720e-468f-4180-9e47-9dce2ec0f1c8","Type":"ContainerStarted","Data":"ee508ad786e73e68ea61f47efa275aa18aeca5ae514f5f192c443d9888564c65"} Oct 30 05:53:08.511701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.511603 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:08.513646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.513624 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" event={"ID":"209835cf-0fac-42a7-a57f-a13c76ddcff0","Type":"ContainerStarted","Data":"5fbe7bf87e898e48eb7877d16dd2d601106d93bc1b8f31a7f23b8469c21c0c52"} Oct 30 05:53:08.513765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.513666 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" event={"ID":"209835cf-0fac-42a7-a57f-a13c76ddcff0","Type":"ContainerStarted","Data":"7b4f01f3536cc0949a73f033a7e25395306efd361410bc09891bf2ace9031f4d"} Oct 30 05:53:08.513765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.513708 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:08.559457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.559381 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" podStartSLOduration=2.559369974 podStartE2EDuration="2.559369974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:08.557006441 +0000 UTC m=+1246.997383634" watchObservedRunningTime="2026-10-30 05:53:08.559369974 +0000 UTC m=+1246.999747165" Oct 30 05:53:08.600874 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:08.600825 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" podStartSLOduration=1.60081271 podStartE2EDuration="1.60081271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:08.59748722 +0000 UTC m=+1247.037864413" watchObservedRunningTime="2026-10-30 05:53:08.60081271 +0000 UTC m=+1247.041189897" Oct 30 05:53:13.120069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:13.115556 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:53:19.543558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:19.543528 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 05:53:19.569065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:19.566443 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 05:53:21.909058 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.909009 6171 scope.go:122] "RemoveContainer" containerID="069837a22f0b80fb6f44be855a0cb97197047a26abefc9a52cde5dee59207369" Oct 30 05:53:21.921613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.921583 6171 scope.go:122] "RemoveContainer" containerID="512c9b22f77600f55aaa4bf2c0a36f4f7b2308a4bfaddc66ff38a320d4242ced" Oct 30 05:53:21.934430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.934410 6171 scope.go:122] "RemoveContainer" containerID="000fba00a7019259d25c1afc1fda15881dfcf34fab8286a5544a7ba548175eb9" Oct 30 05:53:21.945194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.945169 6171 scope.go:122] "RemoveContainer" containerID="65a92e7972ef8e832edef10e1a28b7a5876414c60f83db080cd99a21ebba040a" Oct 30 05:53:21.957111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.957086 6171 scope.go:122] "RemoveContainer" containerID="fc568c0fe26e496543e6e2523a170c18e5f34ff35a00bb8ad835c3f5b479035a" Oct 30 05:53:21.978098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:21.978025 6171 scope.go:122] "RemoveContainer" containerID="c1f1d5353ce692fef76aa24bdf0836a38ece49aa43cec8493aff425662803ef7" Oct 30 05:53:24.755409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:24.755368 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq"] Oct 30 05:53:24.984135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:24.980152 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.029739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.029655 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq"] Oct 30 05:53:25.029910 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.029656 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:53:25.041486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.041451 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:53:25.041486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.041469 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:53:25.041664 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.041532 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:53:25.115885 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.115851 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.116086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.115972 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4htdf\" (UniqueName: \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.116086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.116006 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.217420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.217382 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.217598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.217468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4htdf\" (UniqueName: \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.217598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.217498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.218100 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.218081 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.219999 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.219976 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.240646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.240625 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4htdf\" (UniqueName: \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\") pod \"maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.349117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.344201 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:25.619277 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.619197 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx"] Oct 30 05:53:25.633431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.633394 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.638088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.638060 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx"] Oct 30 05:53:25.652747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.652723 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.653133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.653110 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xvnm\" (UniqueName: \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.653223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.653181 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.754812 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.754781 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.755135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.755112 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xvnm\" (UniqueName: \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.755278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.755200 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.755594 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.755573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.757684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.757664 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.821075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.811874 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xvnm\" (UniqueName: \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\") pod \"maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:25.901534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.888690 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq"] Oct 30 05:53:25.904327 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:25.902874 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca1938a_9ea4_4a87_964a_800e592e7274.slice/crio-82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23 WatchSource:0}: Error finding container 82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23: Status 404 returned error can't find the container with id 82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23 Oct 30 05:53:25.958820 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:25.958795 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:26.062391 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:26.062357 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" event={"ID":"4ca1938a-9ea4-4a87-964a-800e592e7274","Type":"ContainerStarted","Data":"82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23"} Oct 30 05:53:26.284940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:26.284915 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx"] Oct 30 05:53:26.384984 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:26.384954 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:53:27.074180 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:27.074147 6171 generic.go:356] "Generic (PLEG): container finished" podID="4ca1938a-9ea4-4a87-964a-800e592e7274" containerID="8090a867c2d2e22def637580ea2d05f698d1f4b82eb3df7358f57fb4a20f3b5f" exitCode=0 Oct 30 05:53:27.074625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:27.074235 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" event={"ID":"4ca1938a-9ea4-4a87-964a-800e592e7274","Type":"ContainerDied","Data":"8090a867c2d2e22def637580ea2d05f698d1f4b82eb3df7358f57fb4a20f3b5f"} Oct 30 05:53:27.076095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:27.076072 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" event={"ID":"08c3d536-f73e-4cfe-a0ca-909e401e5f6d","Type":"ContainerStarted","Data":"5ef2be1bbb54f6ffe5f25cf90b150550e63639a0f2e1b84e523f0c74cc15ac71"} Oct 30 05:53:27.076215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:27.076117 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" event={"ID":"08c3d536-f73e-4cfe-a0ca-909e401e5f6d","Type":"ContainerStarted","Data":"9c8117a160d780c7589e53222e16c748526b4488ef68f7550c4b32e96b18505b"} Oct 30 05:53:27.170353 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:27.170259 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" podStartSLOduration=2.170245565 podStartE2EDuration="2.170245565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:27.166535332 +0000 UTC m=+1265.606912525" watchObservedRunningTime="2026-10-30 05:53:27.170245565 +0000 UTC m=+1265.610622754" Oct 30 05:53:28.088536 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.088496 6171 generic.go:356] "Generic (PLEG): container finished" podID="08c3d536-f73e-4cfe-a0ca-909e401e5f6d" containerID="5ef2be1bbb54f6ffe5f25cf90b150550e63639a0f2e1b84e523f0c74cc15ac71" exitCode=0 Oct 30 05:53:28.089025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.088592 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" event={"ID":"08c3d536-f73e-4cfe-a0ca-909e401e5f6d","Type":"ContainerDied","Data":"5ef2be1bbb54f6ffe5f25cf90b150550e63639a0f2e1b84e523f0c74cc15ac71"} Oct 30 05:53:28.231390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.231207 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:28.231390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.231283 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:28.323607 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.323577 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:28.421692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.421589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\") pod \"4ca1938a-9ea4-4a87-964a-800e592e7274\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " Oct 30 05:53:28.421872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.421730 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\" (UniqueName: \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\") pod \"4ca1938a-9ea4-4a87-964a-800e592e7274\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " Oct 30 05:53:28.422031 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.422011 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\") pod \"4ca1938a-9ea4-4a87-964a-800e592e7274\" (UID: \"4ca1938a-9ea4-4a87-964a-800e592e7274\") " Oct 30 05:53:28.422836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.422531 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca" pod "4ca1938a-9ea4-4a87-964a-800e592e7274" (UID: "4ca1938a-9ea4-4a87-964a-800e592e7274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:28.422836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.422805 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:28.424626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.424597 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf" pod "4ca1938a-9ea4-4a87-964a-800e592e7274" (UID: "4ca1938a-9ea4-4a87-964a-800e592e7274"). InnerVolumeSpecName "kube-api-access-4htdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:28.424733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.424696 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca" pod "4ca1938a-9ea4-4a87-964a-800e592e7274" (UID: "4ca1938a-9ea4-4a87-964a-800e592e7274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:28.524779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.524751 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ca1938a-9ea4-4a87-964a-800e592e7274-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:28.524779 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:28.524777 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4htdf\" (UniqueName: \"kubernetes.io/projected/4ca1938a-9ea4-4a87-964a-800e592e7274-kube-api-access-4htdf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:29.152488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.152443 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" Oct 30 05:53:29.171088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.165875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq" event={"ID":"4ca1938a-9ea4-4a87-964a-800e592e7274","Type":"ContainerDied","Data":"82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23"} Oct 30 05:53:29.171088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.165943 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82eccd860b7379a2d56a4dc0227aba79d36ac3d9730f64038d0d82ed620d1f23" Oct 30 05:53:29.365307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.365280 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:29.426273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.426203 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\") pod \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " Oct 30 05:53:29.426479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.426458 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\" (UniqueName: \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\") pod \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " Oct 30 05:53:29.426528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.426508 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\") pod \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\" (UID: \"08c3d536-f73e-4cfe-a0ca-909e401e5f6d\") " Oct 30 05:53:29.427786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.427756 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca" pod "08c3d536-f73e-4cfe-a0ca-909e401e5f6d" (UID: "08c3d536-f73e-4cfe-a0ca-909e401e5f6d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:29.428825 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.428796 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca" pod "08c3d536-f73e-4cfe-a0ca-909e401e5f6d" (UID: "08c3d536-f73e-4cfe-a0ca-909e401e5f6d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:29.428968 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.428843 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm" pod "08c3d536-f73e-4cfe-a0ca-909e401e5f6d" (UID: "08c3d536-f73e-4cfe-a0ca-909e401e5f6d"). InnerVolumeSpecName "kube-api-access-4xvnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:29.528935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.528844 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:29.528935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.528870 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xvnm\" (UniqueName: \"kubernetes.io/projected/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-kube-api-access-4xvnm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:29.528935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.528881 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08c3d536-f73e-4cfe-a0ca-909e401e5f6d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:29.911304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.911263 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:53:29.912707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.912675 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4ca1938a-9ea4-4a87-964a-800e592e7274" containerName="revoke-keys" Oct 30 05:53:29.912707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.912705 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ca1938a-9ea4-4a87-964a-800e592e7274" containerName="revoke-keys" Oct 30 05:53:29.912951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.912758 6171 cpu_manager.go:414] "containerMap: removing container" podUID="08c3d536-f73e-4cfe-a0ca-909e401e5f6d" containerName="revoke-keys" Oct 30 05:53:29.912951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.912766 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c3d536-f73e-4cfe-a0ca-909e401e5f6d" containerName="revoke-keys" Oct 30 05:53:29.913084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.912994 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ca1938a-9ea4-4a87-964a-800e592e7274" containerName="revoke-keys" Oct 30 05:53:29.913084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.913018 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c3d536-f73e-4cfe-a0ca-909e401e5f6d" containerName="revoke-keys" Oct 30 05:53:29.918648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.918622 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:29.938551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.938492 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:53:29.939695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:29.939676 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-1b2243ef-openshift-default-dockercfg-jvrmz\"" Oct 30 05:53:30.036027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036001 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036171 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjgp7\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036197 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036307 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036337 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036361 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036494 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036558 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.039773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.036748 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138267 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138430 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138533 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjgp7\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138569 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138649 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138686 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138715 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138753 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.138810 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.139622 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.161072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.139848 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.168069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.162945 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.170473 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.169988 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.170782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.170740 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.228975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.226730 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.228975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.227360 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.228975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.227878 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.291947 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.291855 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" event={"ID":"08c3d536-f73e-4cfe-a0ca-909e401e5f6d","Type":"ContainerDied","Data":"9c8117a160d780c7589e53222e16c748526b4488ef68f7550c4b32e96b18505b"} Oct 30 05:53:30.292432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.292412 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c8117a160d780c7589e53222e16c748526b4488ef68f7550c4b32e96b18505b" Oct 30 05:53:30.292585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.292229 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx" Oct 30 05:53:30.301722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.301620 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjgp7\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.552522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.552311 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:30.938765 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:30.938735 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podace697a1_bb27_4c18_a1e7_3e3ba57dfd32.slice/crio-2dd180ce77031886a307ffb8426c9822d1f14371a2d6f337ea862e82855f772b WatchSource:0}: Error finding container 2dd180ce77031886a307ffb8426c9822d1f14371a2d6f337ea862e82855f772b: Status 404 returned error can't find the container with id 2dd180ce77031886a307ffb8426c9822d1f14371a2d6f337ea862e82855f772b Oct 30 05:53:30.942171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.942130 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:30.942298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.942222 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:30.942374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.942296 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:30.942374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:30.942313 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:53:31.305493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.305454 6171 generic.go:356] "Generic (PLEG): container finished" podID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerID="ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593" exitCode=0 Oct 30 05:53:31.305978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.305531 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerDied","Data":"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593"} Oct 30 05:53:31.306350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.306329 6171 scope.go:122] "RemoveContainer" containerID="ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593" Oct 30 05:53:31.308380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.308346 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:31.308529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.308416 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerStarted","Data":"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f"} Oct 30 05:53:31.308529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.308432 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:31.308529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.308459 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerStarted","Data":"2dd180ce77031886a307ffb8426c9822d1f14371a2d6f337ea862e82855f772b"} Oct 30 05:53:31.308529 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.308469 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:31.440684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.440613 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podStartSLOduration=2.4405941970000002 podStartE2EDuration="2.440594197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:31.434328441 +0000 UTC m=+1269.874705634" watchObservedRunningTime="2026-10-30 05:53:31.440594197 +0000 UTC m=+1269.880971392" Oct 30 05:53:31.565134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:31.562090 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:32.320349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.320316 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerStarted","Data":"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158"} Oct 30 05:53:32.579801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.577352 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:32.605076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.604097 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl"] Oct 30 05:53:32.653223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.653193 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.656902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.656502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.656902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.656630 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj9ds\" (UniqueName: \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.656902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.656659 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.692427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.690130 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:53:32.692427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.690853 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:53:32.692427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.690925 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:53:32.692427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.690974 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:53:32.783576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.758425 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj9ds\" (UniqueName: \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.783576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.758477 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.783576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.758859 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.783576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.760993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.783576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.776111 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl"] Oct 30 05:53:32.823070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.822630 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj9ds\" (UniqueName: \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:32.825173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:32.825020 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d8a893-xptcl\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:33.045490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.045453 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:33.045755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.045731 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.045880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.045777 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:33.102067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.101293 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:33.333927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.333841 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:33.334765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.334747 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:53:33.448979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:33.444633 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl"] Oct 30 05:53:34.045624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:34.045593 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.045807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:34.045646 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:34.345102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:34.344986 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" event={"ID":"7fc29184-99af-4928-8b56-472bbaa7c1f9","Type":"ContainerStarted","Data":"f6214f98eebf8e73b93e14fc81d5e3025c6e8260ab6f1df02a57d0c3d530e9ca"} Oct 30 05:53:34.345102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:34.345036 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" event={"ID":"7fc29184-99af-4928-8b56-472bbaa7c1f9","Type":"ContainerStarted","Data":"37fe7b62c6c6b8863bc790efcd8059ee5a566e786ca44b031597fdc07958b4b3"} Oct 30 05:53:34.407845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:34.407774 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" podStartSLOduration=2.40775743 podStartE2EDuration="2.40775743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:34.405981753 +0000 UTC m=+1272.846358947" watchObservedRunningTime="2026-10-30 05:53:34.40775743 +0000 UTC m=+1272.848134623" Oct 30 05:53:35.047411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.045648 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.047411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.045719 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:35.277814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.277780 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:53:35.311565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.311484 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:53:35.360996 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.360959 6171 generic.go:356] "Generic (PLEG): container finished" podID="7fc29184-99af-4928-8b56-472bbaa7c1f9" containerID="f6214f98eebf8e73b93e14fc81d5e3025c6e8260ab6f1df02a57d0c3d530e9ca" exitCode=0 Oct 30 05:53:35.362230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:35.362201 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" event={"ID":"7fc29184-99af-4928-8b56-472bbaa7c1f9","Type":"ContainerDied","Data":"f6214f98eebf8e73b93e14fc81d5e3025c6e8260ab6f1df02a57d0c3d530e9ca"} Oct 30 05:53:36.045385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:36.045354 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.045565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:36.045409 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:36.742464 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:36.742435 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:37.046938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.045783 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.046938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.045836 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:37.095672 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.095636 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\") pod \"7fc29184-99af-4928-8b56-472bbaa7c1f9\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " Oct 30 05:53:37.096127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.096100 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\" (UniqueName: \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\") pod \"7fc29184-99af-4928-8b56-472bbaa7c1f9\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " Oct 30 05:53:37.096354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.096338 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\") pod \"7fc29184-99af-4928-8b56-472bbaa7c1f9\" (UID: \"7fc29184-99af-4928-8b56-472bbaa7c1f9\") " Oct 30 05:53:37.097776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.097753 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca" pod "7fc29184-99af-4928-8b56-472bbaa7c1f9" (UID: "7fc29184-99af-4928-8b56-472bbaa7c1f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:37.108156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.108126 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca" pod "7fc29184-99af-4928-8b56-472bbaa7c1f9" (UID: "7fc29184-99af-4928-8b56-472bbaa7c1f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:37.108849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.108819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds" pod "7fc29184-99af-4928-8b56-472bbaa7c1f9" (UID: "7fc29184-99af-4928-8b56-472bbaa7c1f9"). InnerVolumeSpecName "kube-api-access-gj9ds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:37.205071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.204571 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:37.205328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.205274 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj9ds\" (UniqueName: \"kubernetes.io/projected/7fc29184-99af-4928-8b56-472bbaa7c1f9-kube-api-access-gj9ds\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:37.205328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.205314 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fc29184-99af-4928-8b56-472bbaa7c1f9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:37.692871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.691507 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" Oct 30 05:53:37.721993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.721956 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl" event={"ID":"7fc29184-99af-4928-8b56-472bbaa7c1f9","Type":"ContainerDied","Data":"37fe7b62c6c6b8863bc790efcd8059ee5a566e786ca44b031597fdc07958b4b3"} Oct 30 05:53:37.722194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:37.722017 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37fe7b62c6c6b8863bc790efcd8059ee5a566e786ca44b031597fdc07958b4b3" Oct 30 05:53:38.045226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:38.045185 6171 patch_prober.go:29] interesting pod/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.045608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:38.045245 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 05:53:39.066986 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.066952 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:39.079284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.078214 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:39.080212 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.080190 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:39.403902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.403821 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 05:53:39.404687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.404669 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7fc29184-99af-4928-8b56-472bbaa7c1f9" containerName="revoke-keys" Oct 30 05:53:39.404783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.404688 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc29184-99af-4928-8b56-472bbaa7c1f9" containerName="revoke-keys" Oct 30 05:53:39.404839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.404828 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc29184-99af-4928-8b56-472bbaa7c1f9" containerName="revoke-keys" Oct 30 05:53:39.414492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.414469 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.453971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.453948 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-477f6b-openshift-default-dockercfg-q6lhv\"" Oct 30 05:53:39.462622 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.462595 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 05:53:39.578612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578577 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578628 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578689 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578813 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578857 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578879 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.578994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.578907 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr5wh\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.579099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.579075 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.579173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.579161 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682318 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682441 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682502 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682658 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682700 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682778 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr5wh\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.682939 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691117 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.690856 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.691748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.691656 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.695067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.691887 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.695067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.693177 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.695067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.694867 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.803834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.766649 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr5wh\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.832116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.823033 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.867173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.846732 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:39.903537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:39.903495 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\") pod \"e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:40.031366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.031334 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:40.280196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.278934 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" podUID="caf2dd47-05a9-41a7-8832-02a30fe858b4" containerName="payload-processing" containerID="cri-o://fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183" gracePeriod=25 Oct 30 05:53:40.312662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.312128 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" podUID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" containerName="payload-pre-processing" containerID="cri-o://3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee" gracePeriod=25 Oct 30 05:53:40.573187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.572234 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:53:40.580075 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:40.580025 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961d6e36_ecec_4957_abdc_76ea72000259.slice/crio-655f505cad4d9176d16ca9357ee52bd47763f8f737557042cc7d697af95c4308 WatchSource:0}: Error finding container 655f505cad4d9176d16ca9357ee52bd47763f8f737557042cc7d697af95c4308: Status 404 returned error can't find the container with id 655f505cad4d9176d16ca9357ee52bd47763f8f737557042cc7d697af95c4308 Oct 30 05:53:40.583112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.583029 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:40.583203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.583149 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:40.583203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.583191 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:53:40.590295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.590269 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 05:53:40.636107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.636084 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:53:40.736978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.736835 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\") pod \"caf2dd47-05a9-41a7-8832-02a30fe858b4\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " Oct 30 05:53:40.736978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.736894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\" (UniqueName: \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\") pod \"caf2dd47-05a9-41a7-8832-02a30fe858b4\" (UID: \"caf2dd47-05a9-41a7-8832-02a30fe858b4\") " Oct 30 05:53:40.737205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.736994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\") pod \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " Oct 30 05:53:40.737205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.737125 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\" (UniqueName: \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\") pod \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\" (UID: \"a1b30fa8-581a-4a98-ad52-e6e5a2170b82\") " Oct 30 05:53:40.738375 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.738182 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume" pod "caf2dd47-05a9-41a7-8832-02a30fe858b4" (UID: "caf2dd47-05a9-41a7-8832-02a30fe858b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:40.738551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.738518 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume" pod "a1b30fa8-581a-4a98-ad52-e6e5a2170b82" (UID: "a1b30fa8-581a-4a98-ad52-e6e5a2170b82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:40.741131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.741093 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz" pod "a1b30fa8-581a-4a98-ad52-e6e5a2170b82" (UID: "a1b30fa8-581a-4a98-ad52-e6e5a2170b82"). InnerVolumeSpecName "kube-api-access-xqsjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:40.741644 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.741608 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t" pod "caf2dd47-05a9-41a7-8832-02a30fe858b4" (UID: "caf2dd47-05a9-41a7-8832-02a30fe858b4"). InnerVolumeSpecName "kube-api-access-cfj2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:40.840606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.840518 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf2dd47-05a9-41a7-8832-02a30fe858b4-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:40.840606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.840551 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfj2t\" (UniqueName: \"kubernetes.io/projected/caf2dd47-05a9-41a7-8832-02a30fe858b4-kube-api-access-cfj2t\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:40.840606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.840566 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:40.840606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.840579 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqsjz\" (UniqueName: \"kubernetes.io/projected/a1b30fa8-581a-4a98-ad52-e6e5a2170b82-kube-api-access-xqsjz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:40.973951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.973920 6171 generic.go:356] "Generic (PLEG): container finished" podID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" containerID="3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee" exitCode=0 Oct 30 05:53:40.974131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.973972 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" Oct 30 05:53:40.974131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.974004 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" event={"ID":"a1b30fa8-581a-4a98-ad52-e6e5a2170b82","Type":"ContainerDied","Data":"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee"} Oct 30 05:53:40.974131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.974104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq" event={"ID":"a1b30fa8-581a-4a98-ad52-e6e5a2170b82","Type":"ContainerDied","Data":"51e27d1d7b1fca089929e5b2bc5886cb8486fe9d92a22b2f86ccd046d9cf43bd"} Oct 30 05:53:40.974336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.974134 6171 scope.go:122] "RemoveContainer" containerID="3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee" Oct 30 05:53:40.976037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.976014 6171 generic.go:356] "Generic (PLEG): container finished" podID="caf2dd47-05a9-41a7-8832-02a30fe858b4" containerID="fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183" exitCode=0 Oct 30 05:53:40.976166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.976087 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" event={"ID":"caf2dd47-05a9-41a7-8832-02a30fe858b4","Type":"ContainerDied","Data":"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183"} Oct 30 05:53:40.976166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.976138 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" event={"ID":"caf2dd47-05a9-41a7-8832-02a30fe858b4","Type":"ContainerDied","Data":"034104d600abcec8721442ee20457dee70a29a353dbe9c97e0747c04fba12ade"} Oct 30 05:53:40.976166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.976139 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh" Oct 30 05:53:40.986136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.986111 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerStarted","Data":"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008"} Oct 30 05:53:40.986226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.986164 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerStarted","Data":"655f505cad4d9176d16ca9357ee52bd47763f8f737557042cc7d697af95c4308"} Oct 30 05:53:40.987660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.987637 6171 scope.go:122] "RemoveContainer" containerID="3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee" Oct 30 05:53:40.988033 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:40.987925 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee\": container with ID starting with 3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee not found: ID does not exist" containerID="3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee" Oct 30 05:53:40.988033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.987957 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee"} err="failed to get container status \"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee\": rpc error: code = NotFound desc = could not find container \"3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee\": container with ID starting with 3a09d8733345b4f376dedd4db1235415f82710d4c903ee43a7e704f7381ec0ee not found: ID does not exist" Oct 30 05:53:40.988033 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:40.987980 6171 scope.go:122] "RemoveContainer" containerID="fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183" Oct 30 05:53:41.001420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.001389 6171 scope.go:122] "RemoveContainer" containerID="fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183" Oct 30 05:53:41.001674 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:41.001654 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183\": container with ID starting with fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183 not found: ID does not exist" containerID="fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183" Oct 30 05:53:41.001722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.001682 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183"} err="failed to get container status \"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183\": rpc error: code = NotFound desc = could not find container \"fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183\": container with ID starting with fa13448179261137215ca42550c7b8af9864dd4f3fdd8839d5249098c8c50183 not found: ID does not exist" Oct 30 05:53:41.032250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.032231 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:41.033585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.033561 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:53:41.033691 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.033608 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:53:41.043130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.043105 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:53:41.052294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.052274 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-a095fc35-54d6b699-9brgh"] Oct 30 05:53:41.157124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.150284 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podStartSLOduration=2.150263752 podStartE2EDuration="2.150263752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:41.104483963 +0000 UTC m=+1279.544861164" watchObservedRunningTime="2026-10-30 05:53:41.150263752 +0000 UTC m=+1279.590640945" Oct 30 05:53:41.201080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.192087 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:53:41.212077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.207063 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-a095fc35-5b9f5c95c9-d77dq"] Oct 30 05:53:41.834082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.833207 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" path="/var/lib/kubelet/pods/a1b30fa8-581a-4a98-ad52-e6e5a2170b82/volumes" Oct 30 05:53:41.851072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.848657 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf2dd47-05a9-41a7-8832-02a30fe858b4" path="/var/lib/kubelet/pods/caf2dd47-05a9-41a7-8832-02a30fe858b4/volumes" Oct 30 05:53:41.932721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:41.928227 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq"] Oct 30 05:53:42.030072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:42.023815 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-a095fc35-v4vnq"] Oct 30 05:53:42.071722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:42.066829 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:42.079387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:42.079363 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:42.080613 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:42.080593 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:53:43.321497 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:43.321460 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:53:43.322008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:43.321757 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" containerID="cri-o://5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158" gracePeriod=30 Oct 30 05:53:43.739087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:43.738596 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ca1938a-9ea4-4a87-964a-800e592e7274" path="/var/lib/kubelet/pods/4ca1938a-9ea4-4a87-964a-800e592e7274/volumes" Oct 30 05:53:45.428747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:45.405780 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx"] Oct 30 05:53:45.453318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:45.453286 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-e2a1565f-tr4qx"] Oct 30 05:53:45.878144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:45.878105 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c3d536-f73e-4cfe-a0ca-909e401e5f6d" path="/var/lib/kubelet/pods/08c3d536-f73e-4cfe-a0ca-909e401e5f6d/volumes" Oct 30 05:53:47.334247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:47.334216 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:53:48.613998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.613939 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:48.783066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.782723 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.782867 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.782943 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783011 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783089 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783212 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783239 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783268 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.783416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.783316 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:48.870159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.869257 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:48.870159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.869470 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:48.870159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.869877 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:48.898720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.878756 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:48.941390 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.939155 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:48.983588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.983526 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:49.000083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:48.983937 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:53:49.000634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.000572 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "kube-api-access-fq4wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.001399 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") pod \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\" (UID: \"52e3f15a-e805-4e4c-b81d-49b0548e1d19\") " Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002520 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002547 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002562 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002575 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002587 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002599 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002610 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52e3f15a-e805-4e4c-b81d-49b0548e1d19-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:49.002700 6171 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/52e3f15a-e805-4e4c-b81d-49b0548e1d19/volumes/kubernetes.io~projected/kube-api-access-fq4wx Oct 30 05:53:49.002775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.002746 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "kube-api-access-fq4wx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:49.012474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.012432 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo" pod "52e3f15a-e805-4e4c-b81d-49b0548e1d19" (UID: "52e3f15a-e805-4e4c-b81d-49b0548e1d19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:53:49.143475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.142558 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52e3f15a-e805-4e4c-b81d-49b0548e1d19-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.143475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.142590 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4wx\" (UniqueName: \"kubernetes.io/projected/52e3f15a-e805-4e4c-b81d-49b0548e1d19-kube-api-access-fq4wx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:53:49.416396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.413965 6171 generic.go:356] "Generic (PLEG): container finished" podID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerID="5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158" exitCode=0 Oct 30 05:53:49.416396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.414192 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerDied","Data":"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158"} Oct 30 05:53:49.416396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.414251 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" event={"ID":"52e3f15a-e805-4e4c-b81d-49b0548e1d19","Type":"ContainerDied","Data":"9542d3f4595b0604c8e8fbb1b404923bcd1320a6a07e46bb2b8e77fb5c6014c3"} Oct 30 05:53:49.416396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.414281 6171 scope.go:122] "RemoveContainer" containerID="5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158" Oct 30 05:53:49.416396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.414509 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548" Oct 30 05:53:49.441357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.441329 6171 scope.go:122] "RemoveContainer" containerID="ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593" Oct 30 05:53:49.452624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.452604 6171 scope.go:122] "RemoveContainer" containerID="5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158" Oct 30 05:53:49.452916 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:49.452898 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158\": container with ID starting with 5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158 not found: ID does not exist" containerID="5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158" Oct 30 05:53:49.452976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.452923 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158"} err="failed to get container status \"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158\": rpc error: code = NotFound desc = could not find container \"5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158\": container with ID starting with 5b717cfbec02791dc13fbc84865bc1342a357a141cdccceb161d4f7e60bd5158 not found: ID does not exist" Oct 30 05:53:49.452976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.452938 6171 scope.go:122] "RemoveContainer" containerID="ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593" Oct 30 05:53:49.453264 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:49.453241 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593\": container with ID starting with ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593 not found: ID does not exist" containerID="ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593" Oct 30 05:53:49.453315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.453276 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593"} err="failed to get container status \"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593\": rpc error: code = NotFound desc = could not find container \"ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593\": container with ID starting with ab106c57d1342cea765a14781abcc722ad38a965f32c6105503ae5b1be250593 not found: ID does not exist" Oct 30 05:53:49.489792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.489771 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:53:49.501815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.501792 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a095fc35-openshift-default-574ccf8d8f-bz548"] Oct 30 05:53:49.777736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:49.764734 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" path="/var/lib/kubelet/pods/52e3f15a-e805-4e4c-b81d-49b0548e1d19/volumes" Oct 30 05:53:51.749546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.749499 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 05:53:51.750311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750295 6171 cpu_manager.go:414] "containerMap: removing container" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.750311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750311 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.750428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750360 6171 cpu_manager.go:414] "containerMap: removing container" podUID="caf2dd47-05a9-41a7-8832-02a30fe858b4" containerName="payload-processing" Oct 30 05:53:51.750428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750365 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf2dd47-05a9-41a7-8832-02a30fe858b4" containerName="payload-processing" Oct 30 05:53:51.750428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750383 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" containerName="payload-pre-processing" Oct 30 05:53:51.750428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750388 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" containerName="payload-pre-processing" Oct 30 05:53:51.750553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750498 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.750553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750509 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf2dd47-05a9-41a7-8832-02a30fe858b4" containerName="payload-processing" Oct 30 05:53:51.750553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750516 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1b30fa8-581a-4a98-ad52-e6e5a2170b82" containerName="payload-pre-processing" Oct 30 05:53:51.750553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.750524 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.756739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.756721 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:51.804160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.803247 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-815776-dockercfg-8hbhn\"" Oct 30 05:53:51.804160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.803557 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-815776\"" Oct 30 05:53:51.821372 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.821338 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 05:53:51.830153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.830127 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9207-cc6f8d"] Oct 30 05:53:51.831021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.830999 6171 cpu_manager.go:414] "containerMap: removing container" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.831021 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.831020 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52e3f15a-e805-4e4c-b81d-49b0548e1d19" containerName="istio-proxy" Oct 30 05:53:51.835855 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.835834 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:53:51.848990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.848965 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-cc6f8d"] Oct 30 05:53:51.872783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.872747 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 05:53:51.879184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.879122 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:51.911435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.911411 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4m56v\"" Oct 30 05:53:51.919457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.919423 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chtb8\" (UniqueName: \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\") pod \"test-curl-9207-cc6f8d\" (UID: \"cdd5b87a-afdf-4bd4-b579-81cc7f642000\") " pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:53:51.919621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.919482 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:51.919800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.919778 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btgzt\" (UniqueName: \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:51.928150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:51.928127 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 05:53:52.085059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.078113 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btgzt\" (UniqueName: \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:52.085059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.078245 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfj6m\" (UniqueName: \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.085059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.078428 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chtb8\" (UniqueName: \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\") pod \"test-curl-9207-cc6f8d\" (UID: \"cdd5b87a-afdf-4bd4-b579-81cc7f642000\") " pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:53:52.085059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.078473 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.085059 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.078513 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:52.224183 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.224133 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btgzt\" (UniqueName: \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:52.238621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.238582 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:52.304735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.304700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chtb8\" (UniqueName: \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\") pod \"test-curl-9207-cc6f8d\" (UID: \"cdd5b87a-afdf-4bd4-b579-81cc7f642000\") " pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:53:52.314470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.314432 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfj6m\" (UniqueName: \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.314967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.314938 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.316328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.316305 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.374118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.366177 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfj6m\" (UniqueName: \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\") pod \"payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:52.374806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.374697 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:52.459217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.455335 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:53:52.534111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:52.531389 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:53.010414 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.010382 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 05:53:53.011424 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:53.011401 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890c1245_89a6_4e57_a78a_455e5aab5352.slice/crio-8c1293bb915fffb6d2d24ddd9a9cdfcba3196880b7fd2df8df4fe6875c6ab9fc WatchSource:0}: Error finding container 8c1293bb915fffb6d2d24ddd9a9cdfcba3196880b7fd2df8df4fe6875c6ab9fc: Status 404 returned error can't find the container with id 8c1293bb915fffb6d2d24ddd9a9cdfcba3196880b7fd2df8df4fe6875c6ab9fc Oct 30 05:53:53.024585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.024564 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-cc6f8d"] Oct 30 05:53:53.025763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.025747 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:53:53.117346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.117320 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 05:53:53.649717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.649678 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-cc6f8d" event={"ID":"cdd5b87a-afdf-4bd4-b579-81cc7f642000","Type":"ContainerStarted","Data":"ccf9cf48f97c25b95fbcf9cf05cd630872c8553f89bf92fcab8a6a713ab1b643"} Oct 30 05:53:53.651502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.651477 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" event={"ID":"1dfd15d8-431d-44f1-835d-83e348a45fb9","Type":"ContainerStarted","Data":"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0"} Oct 30 05:53:53.651643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.651523 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" event={"ID":"1dfd15d8-431d-44f1-835d-83e348a45fb9","Type":"ContainerStarted","Data":"4beb2a5b259bf008f39ad44e00b6abcf1df1f0b4c69543e422045160231e7022"} Oct 30 05:53:53.651643 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.651558 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:53:53.653582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.653552 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" event={"ID":"890c1245-89a6-4e57-a78a-455e5aab5352","Type":"ContainerStarted","Data":"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e"} Oct 30 05:53:53.653667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.653598 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" event={"ID":"890c1245-89a6-4e57-a78a-455e5aab5352","Type":"ContainerStarted","Data":"8c1293bb915fffb6d2d24ddd9a9cdfcba3196880b7fd2df8df4fe6875c6ab9fc"} Oct 30 05:53:53.653667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.653622 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:53:53.718663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.718603 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" podStartSLOduration=2.718585686 podStartE2EDuration="2.718585686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:53.716498202 +0000 UTC m=+1292.156875395" watchObservedRunningTime="2026-10-30 05:53:53.718585686 +0000 UTC m=+1292.158962882" Oct 30 05:53:53.771193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.771146 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" podStartSLOduration=2.771132096 podStartE2EDuration="2.771132096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:53.765792079 +0000 UTC m=+1292.206169271" watchObservedRunningTime="2026-10-30 05:53:53.771132096 +0000 UTC m=+1292.211509286" Oct 30 05:53:53.956740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.956658 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:53:53.962116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:53.962091 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.011069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.002755 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-3571f8-dockercfg-td9j7\"" Oct 30 05:53:54.011069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.007907 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-3571f8\"" Oct 30 05:53:54.011069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.010609 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:53:54.074068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.071631 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.074068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.071701 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql94c\" (UniqueName: \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.276215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.242557 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.276215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.242624 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql94c\" (UniqueName: \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.276215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.249012 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.455216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.423437 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:53:54.506477 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.477386 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql94c\" (UniqueName: \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\") pod \"payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.622115 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.619327 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:54.653065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.641512 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:53:54.653065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.641609 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.740478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.740428 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r27xh\" (UniqueName: \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.740767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.740537 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.842387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.842347 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r27xh\" (UniqueName: \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.842583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.842423 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.843287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.843264 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:54.935253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:54.935183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r27xh\" (UniqueName: \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\") pod \"payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:55.044441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.044411 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:55.069355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.069329 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:53:55.078101 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:55.077941 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26f0d9e5_89a3_4c65_ae67_d0989a78713e.slice/crio-8316d56ad3e2f7a3353752cdb2f89a5b58cd45f379875b6131e1b3da3d71e508 WatchSource:0}: Error finding container 8316d56ad3e2f7a3353752cdb2f89a5b58cd45f379875b6131e1b3da3d71e508: Status 404 returned error can't find the container with id 8316d56ad3e2f7a3353752cdb2f89a5b58cd45f379875b6131e1b3da3d71e508 Oct 30 05:53:55.432845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.432816 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:53:55.437907 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:53:55.437878 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43e73e7c_e59e_4af0_8678_4d9b82816849.slice/crio-b3a198e61da4aa7c3625f51f5eb27572b03d6e2a955764ce53d0680972e99561 WatchSource:0}: Error finding container b3a198e61da4aa7c3625f51f5eb27572b03d6e2a955764ce53d0680972e99561: Status 404 returned error can't find the container with id b3a198e61da4aa7c3625f51f5eb27572b03d6e2a955764ce53d0680972e99561 Oct 30 05:53:55.703250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.703197 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" event={"ID":"43e73e7c-e59e-4af0-8678-4d9b82816849","Type":"ContainerStarted","Data":"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13"} Oct 30 05:53:55.703443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.703301 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" event={"ID":"43e73e7c-e59e-4af0-8678-4d9b82816849","Type":"ContainerStarted","Data":"b3a198e61da4aa7c3625f51f5eb27572b03d6e2a955764ce53d0680972e99561"} Oct 30 05:53:55.704609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.704586 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:53:55.709073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.708731 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" event={"ID":"26f0d9e5-89a3-4c65-ae67-d0989a78713e","Type":"ContainerStarted","Data":"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712"} Oct 30 05:53:55.709073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.708795 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" event={"ID":"26f0d9e5-89a3-4c65-ae67-d0989a78713e","Type":"ContainerStarted","Data":"8316d56ad3e2f7a3353752cdb2f89a5b58cd45f379875b6131e1b3da3d71e508"} Oct 30 05:53:55.709073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.708831 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:53:55.752602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.752534 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" podStartSLOduration=1.7525154029999999 podStartE2EDuration="1.752515403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:55.747133612 +0000 UTC m=+1294.187510803" watchObservedRunningTime="2026-10-30 05:53:55.752515403 +0000 UTC m=+1294.192892592" Oct 30 05:53:55.800505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:55.800410 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" podStartSLOduration=2.800389431 podStartE2EDuration="2.800389431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:55.792943429 +0000 UTC m=+1294.233320621" watchObservedRunningTime="2026-10-30 05:53:55.800389431 +0000 UTC m=+1294.240766640" Oct 30 05:53:56.389338 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:53:56.389297 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:53:57.733911 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:57.733873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-cc6f8d" event={"ID":"cdd5b87a-afdf-4bd4-b579-81cc7f642000","Type":"ContainerStarted","Data":"40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2"} Oct 30 05:53:57.789252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:57.789185 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9207-cc6f8d" podStartSLOduration=3.137031685 podStartE2EDuration="6.789164816s" totalImagesPullingTime="3.652133131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:57.780830661 +0000 UTC m=+1296.221207853" watchObservedRunningTime="2026-10-30 05:53:57.789164816 +0000 UTC m=+1296.229542008" Oct 30 05:53:58.949569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:58.949488 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9207-cc6f8d"] Oct 30 05:53:59.156735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.156695 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9207-cc6f8d"] Oct 30 05:53:59.223098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.221423 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9207-a414a3"] Oct 30 05:53:59.223098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.222786 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" containerName="test-curl-9207-cc6f8d" Oct 30 05:53:59.223098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.222809 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" containerName="test-curl-9207-cc6f8d" Oct 30 05:53:59.223521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.223501 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" containerName="test-curl-9207-cc6f8d" Oct 30 05:53:59.243834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.243808 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:53:59.335184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.335100 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-a414a3"] Oct 30 05:53:59.369070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.366454 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkd7\" (UniqueName: \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\") pod \"test-curl-9207-a414a3\" (UID: \"8250b2b8-077f-4341-9505-87fe8d1ce3d2\") " pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:53:59.490127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.489341 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkd7\" (UniqueName: \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\") pod \"test-curl-9207-a414a3\" (UID: \"8250b2b8-077f-4341-9505-87fe8d1ce3d2\") " pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:53:59.532630 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.527371 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkd7\" (UniqueName: \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\") pod \"test-curl-9207-a414a3\" (UID: \"8250b2b8-077f-4341-9505-87fe8d1ce3d2\") " pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:53:59.592787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.592760 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:53:59.990101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.984928 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-a414a3"] Oct 30 05:53:59.992882 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:53:59.992849 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9207-cc6f8d" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" containerName="test-curl-9207-cc6f8d" containerID="cri-o://40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2" gracePeriod=2 Oct 30 05:54:00.045255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:00.045220 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g"] Oct 30 05:54:00.055525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:00.055444 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-71ab8229-n277g"] Oct 30 05:54:01.009354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:01.009307 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-a414a3" event={"ID":"8250b2b8-077f-4341-9505-87fe8d1ce3d2","Type":"ContainerStarted","Data":"1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57"} Oct 30 05:54:01.009841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:01.009366 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-a414a3" event={"ID":"8250b2b8-077f-4341-9505-87fe8d1ce3d2","Type":"ContainerStarted","Data":"8830b2c7489d14b0bafaa82ad37a357b5f0d19b43f9a157826ae64988146012c"} Oct 30 05:54:01.066618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:01.066568 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9207-a414a3" podStartSLOduration=3.066552701 podStartE2EDuration="3.066552701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:01.057634971 +0000 UTC m=+1299.498012164" watchObservedRunningTime="2026-10-30 05:54:01.066552701 +0000 UTC m=+1299.506929894" Oct 30 05:54:01.770116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:01.770080 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1826fe-97dc-4c64-919e-031f3c73a714" path="/var/lib/kubelet/pods/af1826fe-97dc-4c64-919e-031f3c73a714/volumes" Oct 30 05:54:02.021925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.021898 6171 generic.go:356] "Generic (PLEG): container finished" podID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerID="56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7" exitCode=0 Oct 30 05:54:02.022335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.021983 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerDied","Data":"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7"} Oct 30 05:54:02.022614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.022596 6171 scope.go:122] "RemoveContainer" containerID="56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7" Oct 30 05:54:02.024586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.024552 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:02.024685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.024659 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:02.024734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.024714 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:02.065457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.065426 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9207-a414a3"] Oct 30 05:54:02.076019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.075944 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9207-a414a3"] Oct 30 05:54:02.229616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.229578 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9207-3d5e36"] Oct 30 05:54:02.233519 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.233491 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" containerName="test-curl-9207-a414a3" Oct 30 05:54:02.233698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.233683 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" containerName="test-curl-9207-a414a3" Oct 30 05:54:02.233981 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.233969 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" containerName="test-curl-9207-a414a3" Oct 30 05:54:02.240006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.239984 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:02.248086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.245672 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-3d5e36"] Oct 30 05:54:02.324726 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.324688 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcftr\" (UniqueName: \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\") pod \"test-curl-9207-3d5e36\" (UID: \"d56ce028-0b33-48a1-a2bd-43fac902f1de\") " pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:02.331668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.331640 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:54:02.426636 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.426606 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\" (UniqueName: \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\") pod \"cdd5b87a-afdf-4bd4-b579-81cc7f642000\" (UID: \"cdd5b87a-afdf-4bd4-b579-81cc7f642000\") " Oct 30 05:54:02.427147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.427127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcftr\" (UniqueName: \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\") pod \"test-curl-9207-3d5e36\" (UID: \"d56ce028-0b33-48a1-a2bd-43fac902f1de\") " pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:02.429396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.429367 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8" pod "cdd5b87a-afdf-4bd4-b579-81cc7f642000" (UID: "cdd5b87a-afdf-4bd4-b579-81cc7f642000"). InnerVolumeSpecName "kube-api-access-chtb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:02.451297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.451274 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcftr\" (UniqueName: \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\") pod \"test-curl-9207-3d5e36\" (UID: \"d56ce028-0b33-48a1-a2bd-43fac902f1de\") " pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:02.528995 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.528954 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chtb8\" (UniqueName: \"kubernetes.io/projected/cdd5b87a-afdf-4bd4-b579-81cc7f642000-kube-api-access-chtb8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:54:02.561690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.561608 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:02.910839 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:02.910811 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd56ce028_0b33_48a1_a2bd_43fac902f1de.slice/crio-e80f8a3431125eaba2024a38f8ae2aedfd9862153442c3c130fd3e0c3a0b60cd WatchSource:0}: Error finding container e80f8a3431125eaba2024a38f8ae2aedfd9862153442c3c130fd3e0c3a0b60cd: Status 404 returned error can't find the container with id e80f8a3431125eaba2024a38f8ae2aedfd9862153442c3c130fd3e0c3a0b60cd Oct 30 05:54:02.912026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:02.911997 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-3d5e36"] Oct 30 05:54:03.033375 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.033331 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5"] Oct 30 05:54:03.035403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.035348 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerStarted","Data":"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf"} Oct 30 05:54:03.036967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.036941 6171 generic.go:356] "Generic (PLEG): container finished" podID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" containerID="40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2" exitCode=137 Oct 30 05:54:03.037219 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.036983 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:54:03.037219 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.037058 6171 scope.go:122] "RemoveContainer" containerID="40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2" Oct 30 05:54:03.039621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.039590 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9207-a414a3" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" containerName="test-curl-9207-a414a3" containerID="cri-o://1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57" gracePeriod=2 Oct 30 05:54:03.039738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.039723 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-3d5e36" event={"ID":"d56ce028-0b33-48a1-a2bd-43fac902f1de","Type":"ContainerStarted","Data":"e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e"} Oct 30 05:54:03.039805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.039752 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-3d5e36" event={"ID":"d56ce028-0b33-48a1-a2bd-43fac902f1de","Type":"ContainerStarted","Data":"e80f8a3431125eaba2024a38f8ae2aedfd9862153442c3c130fd3e0c3a0b60cd"} Oct 30 05:54:03.042155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.042131 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-202061a5-q5mr5"] Oct 30 05:54:03.051073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.051030 6171 scope.go:122] "RemoveContainer" containerID="40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2" Oct 30 05:54:03.051365 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:03.051347 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2\": container with ID starting with 40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2 not found: ID does not exist" containerID="40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2" Oct 30 05:54:03.051431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.051372 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2"} err="failed to get container status \"40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2\": rpc error: code = NotFound desc = could not find container \"40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2\": container with ID starting with 40ee8ce6701c3fea7d68e657d8063b38a12263233e744a1d921e442a26544ab2 not found: ID does not exist" Oct 30 05:54:03.142118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.142036 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9207-3d5e36" podStartSLOduration=1.142022089 podStartE2EDuration="1.142022089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:03.140070903 +0000 UTC m=+1301.580448097" watchObservedRunningTime="2026-10-30 05:54:03.142022089 +0000 UTC m=+1301.582399279" Oct 30 05:54:03.153535 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:03.153507 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-a414a3\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:54:03.166938 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:03.166883 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-cc6f8d\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" pod="opendatahub/test-curl-9207-cc6f8d" Oct 30 05:54:03.361077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.359891 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:54:03.361285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.361221 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:03.361381 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.361274 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:03.380116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.378967 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:54:03.676893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.676859 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbbc142e-6ead-491d-accf-9043427d13ac" path="/var/lib/kubelet/pods/cbbc142e-6ead-491d-accf-9043427d13ac/volumes" Oct 30 05:54:03.677402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:03.677379 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdd5b87a-afdf-4bd4-b579-81cc7f642000" path="/var/lib/kubelet/pods/cdd5b87a-afdf-4bd4-b579-81cc7f642000/volumes" Oct 30 05:54:04.033517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.033484 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8"] Oct 30 05:54:04.039811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.039787 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2bea5546-69mm8"] Oct 30 05:54:04.222593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.222542 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9207-3d5e36"] Oct 30 05:54:04.233297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.233258 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9207-3d5e36"] Oct 30 05:54:04.357821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.357738 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:04.357821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.357805 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:04.366515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.366490 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9207-a42866"] Oct 30 05:54:04.367385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.367368 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" containerName="test-curl-9207-3d5e36" Oct 30 05:54:04.367456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.367386 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" containerName="test-curl-9207-3d5e36" Oct 30 05:54:04.367547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.367538 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" containerName="test-curl-9207-3d5e36" Oct 30 05:54:04.373325 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.373304 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:04.418403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.418366 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-a42866"] Oct 30 05:54:04.477801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.477770 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r97s7\" (UniqueName: \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\") pod \"test-curl-9207-a42866\" (UID: \"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b\") " pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:04.579677 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.579644 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r97s7\" (UniqueName: \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\") pod \"test-curl-9207-a42866\" (UID: \"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b\") " pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:04.607858 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.607791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r97s7\" (UniqueName: \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\") pod \"test-curl-9207-a42866\" (UID: \"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b\") " pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:04.706125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.706097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 05:54:04.708009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.707990 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 05:54:04.718350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:04.718326 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:05.036035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.036003 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt"] Oct 30 05:54:05.042356 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.042332 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60c04f47-phxjt"] Oct 30 05:54:05.068067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.061808 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9207-a42866"] Oct 30 05:54:05.155877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.149640 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9207-3d5e36" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" containerName="test-curl-9207-3d5e36" containerID="cri-o://e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e" gracePeriod=2 Oct 30 05:54:05.185070 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:05.182262 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-3d5e36\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:05.357391 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.357362 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:05.357510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.357411 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:05.517665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.517641 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:54:05.531358 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:05.531333 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-3d5e36\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:05.607596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.607523 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\" (UniqueName: \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\") pod \"8250b2b8-077f-4341-9505-87fe8d1ce3d2\" (UID: \"8250b2b8-077f-4341-9505-87fe8d1ce3d2\") " Oct 30 05:54:05.610243 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.610209 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7" pod "8250b2b8-077f-4341-9505-87fe8d1ce3d2" (UID: "8250b2b8-077f-4341-9505-87fe8d1ce3d2"). InnerVolumeSpecName "kube-api-access-mdkd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:05.675791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.675764 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" path="/var/lib/kubelet/pods/8250b2b8-077f-4341-9505-87fe8d1ce3d2/volumes" Oct 30 05:54:05.676336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.676322 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2e8c1aa-66ee-4271-9057-45d9e5bb580f" path="/var/lib/kubelet/pods/a2e8c1aa-66ee-4271-9057-45d9e5bb580f/volumes" Oct 30 05:54:05.676806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.676792 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a17437-5bde-411e-abe8-08e255ec689b" path="/var/lib/kubelet/pods/c3a17437-5bde-411e-abe8-08e255ec689b/volumes" Oct 30 05:54:05.709715 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:05.709693 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkd7\" (UniqueName: \"kubernetes.io/projected/8250b2b8-077f-4341-9505-87fe8d1ce3d2-kube-api-access-mdkd7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:54:06.166002 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.165962 6171 generic.go:356] "Generic (PLEG): container finished" podID="8250b2b8-077f-4341-9505-87fe8d1ce3d2" containerID="1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57" exitCode=137 Oct 30 05:54:06.166520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.166015 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a414a3" Oct 30 05:54:06.166520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.166099 6171 scope.go:122] "RemoveContainer" containerID="1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57" Oct 30 05:54:06.169166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.169139 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-a42866" event={"ID":"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b","Type":"ContainerStarted","Data":"a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849"} Oct 30 05:54:06.169293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.169202 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9207-a42866" event={"ID":"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b","Type":"ContainerStarted","Data":"d6b46eafca895ddfe7e3f902f077b848bb137a5dd0086bd96daf726b1b715362"} Oct 30 05:54:06.206145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.206120 6171 scope.go:122] "RemoveContainer" containerID="1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57" Oct 30 05:54:06.206595 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:06.206561 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57\": container with ID starting with 1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57 not found: ID does not exist" containerID="1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57" Oct 30 05:54:06.206711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.206607 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57"} err="failed to get container status \"1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57\": rpc error: code = NotFound desc = could not find container \"1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57\": container with ID starting with 1a765365765006259acea6f7a5db43e5baac81f4590b22a49af4eb3f447a4c57 not found: ID does not exist" Oct 30 05:54:06.294072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.286370 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9207-a42866" podStartSLOduration=2.286347759 podStartE2EDuration="2.286347759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:06.233287335 +0000 UTC m=+1304.673664529" watchObservedRunningTime="2026-10-30 05:54:06.286347759 +0000 UTC m=+1304.726724955" Oct 30 05:54:06.357901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.357869 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:06.358091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.357919 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:06.724032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:06.724004 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:54:07.308712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.304869 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9207-a42866"] Oct 30 05:54:07.353073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.344759 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9207-a42866"] Oct 30 05:54:07.363156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.357452 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:07.363156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.357511 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:07.636654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.636634 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:07.737620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.737591 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:54:07.769435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.769347 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\" (UniqueName: \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\") pod \"d56ce028-0b33-48a1-a2bd-43fac902f1de\" (UID: \"d56ce028-0b33-48a1-a2bd-43fac902f1de\") " Oct 30 05:54:07.771768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.771735 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr" pod "d56ce028-0b33-48a1-a2bd-43fac902f1de" (UID: "d56ce028-0b33-48a1-a2bd-43fac902f1de"). InnerVolumeSpecName "kube-api-access-fcftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:07.872992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:07.872958 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcftr\" (UniqueName: \"kubernetes.io/projected/d56ce028-0b33-48a1-a2bd-43fac902f1de-kube-api-access-fcftr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:54:08.344544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.344495 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:54:08.345926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.345902 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" containerName="test-curl-9207-a42866" Oct 30 05:54:08.346085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.345929 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" containerName="test-curl-9207-a42866" Oct 30 05:54:08.346224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.346207 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" containerName="test-curl-9207-a42866" Oct 30 05:54:08.351624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.351599 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.357290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.357264 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:54:08.357411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.357336 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.158:15021/healthz/ready\": dial tcp 10.133.0.158:15021: connect: connection refused" Oct 30 05:54:08.388273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.388240 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d4ccc5-openshift-default-dockercfg-gglz9\"" Oct 30 05:54:08.391004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.389751 6171 generic.go:356] "Generic (PLEG): container finished" podID="d56ce028-0b33-48a1-a2bd-43fac902f1de" containerID="e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e" exitCode=137 Oct 30 05:54:08.391004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.389922 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-3d5e36" Oct 30 05:54:08.391004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.390086 6171 scope.go:122] "RemoveContainer" containerID="e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e" Oct 30 05:54:08.391336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.391151 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9207-a42866" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" containerName="test-curl-9207-a42866" containerID="cri-o://a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849" gracePeriod=2 Oct 30 05:54:08.408213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.408180 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.530025 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.543702 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwr88\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.543833 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.543964 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.543999 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.544082 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.544138 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.544181 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.548068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.544221 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.927123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.927022 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.927459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.927434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932113 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932165 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932270 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932441 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwr88\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932751 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932864 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.932891 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.933577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.933063 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.951080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.939527 6171 scope.go:122] "RemoveContainer" containerID="e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e" Oct 30 05:54:08.951080 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:08.940124 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e\": container with ID starting with e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e not found: ID does not exist" containerID="e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e" Oct 30 05:54:08.951080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.940161 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e"} err="failed to get container status \"e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e\": rpc error: code = NotFound desc = could not find container \"e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e\": container with ID starting with e892be56faa2dcf642a8eaa0045bfed9754e3fe60da1c8e87fc83ebd8e51921e not found: ID does not exist" Oct 30 05:54:08.951080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.948793 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.951080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.950106 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:08.977072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:08.976576 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.034061 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.029468 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.044072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.036882 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.044072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.040696 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.044072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.041387 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwr88\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\") pod \"e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.284069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.283892 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:09.400538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.400501 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:54:09.404104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.404081 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 05:54:09.678351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.678275 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56ce028-0b33-48a1-a2bd-43fac902f1de" path="/var/lib/kubelet/pods/d56ce028-0b33-48a1-a2bd-43fac902f1de/volumes" Oct 30 05:54:09.718216 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:09.718179 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56e9e3ff_400c_4e70_90d4_7efdacaf70ea.slice/crio-7bc2ff9e23da796b53ba367bb24939a2314b81acbb73108413f6ee6f245bf8dc WatchSource:0}: Error finding container 7bc2ff9e23da796b53ba367bb24939a2314b81acbb73108413f6ee6f245bf8dc: Status 404 returned error can't find the container with id 7bc2ff9e23da796b53ba367bb24939a2314b81acbb73108413f6ee6f245bf8dc Oct 30 05:54:09.720874 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.720843 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:09.720945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.720910 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:09.720945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.720941 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:09.722081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:09.722030 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:54:10.431110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.431083 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerStarted","Data":"e91641a98e117442b4a47c23e4fe85a27662f968c13f9acffb125b310691be1b"} Oct 30 05:54:10.431466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.431138 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerStarted","Data":"7bc2ff9e23da796b53ba367bb24939a2314b81acbb73108413f6ee6f245bf8dc"} Oct 30 05:54:10.482035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.481987 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podStartSLOduration=2.48197019 podStartE2EDuration="2.48197019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:10.476272476 +0000 UTC m=+1308.916649683" watchObservedRunningTime="2026-10-30 05:54:10.48197019 +0000 UTC m=+1308.922347383" Oct 30 05:54:10.830134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.828715 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:10.962724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.962690 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\" (UniqueName: \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\") pod \"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b\" (UID: \"e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b\") " Oct 30 05:54:10.965120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:10.965094 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7" pod "e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" (UID: "e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b"). InnerVolumeSpecName "kube-api-access-r97s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:11.064929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.064851 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r97s7\" (UniqueName: \"kubernetes.io/projected/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b-kube-api-access-r97s7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:54:11.290621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.290585 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:11.296516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.296490 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:11.443262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.443177 6171 generic.go:356] "Generic (PLEG): container finished" podID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" containerID="a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849" exitCode=137 Oct 30 05:54:11.443262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.443226 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:11.443734 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.443283 6171 scope.go:122] "RemoveContainer" containerID="a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849" Oct 30 05:54:11.443879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.443857 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:11.445114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.445097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:54:11.456072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.456025 6171 scope.go:122] "RemoveContainer" containerID="a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849" Oct 30 05:54:11.456341 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:11.456318 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849\": container with ID starting with a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849 not found: ID does not exist" containerID="a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849" Oct 30 05:54:11.456444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.456345 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849"} err="failed to get container status \"a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849\": rpc error: code = NotFound desc = could not find container \"a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849\": container with ID starting with a3d374e6dfe458e759bdc35b5103cc33d78455d58522fec99c7360b3b1803849 not found: ID does not exist" Oct 30 05:54:11.506327 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:11.506283 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-a42866\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:11.674519 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:11.674485 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" path="/var/lib/kubelet/pods/e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b/volumes" Oct 30 05:54:11.685956 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:11.685921 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9207-a42866\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="e6b4ce29-3b0b-4dbb-9120-4d04aea6e93b" pod="opendatahub/test-curl-9207-a42866" Oct 30 05:54:14.786321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.786288 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:54:14.792987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.792958 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:14.811750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.811716 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d4ccc5-dockercfg-p7znn\"" Oct 30 05:54:14.818728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.818693 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:54:14.824579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.824559 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d4ccc5\"" Oct 30 05:54:14.844440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.844411 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:54:14.850974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.850950 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:14.881895 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:14.881869 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:54:15.083478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.066074 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.083478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.066249 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspcn\" (UniqueName: \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.083478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.066330 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.083478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.066389 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrjz2\" (UniqueName: \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.205136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.181197 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.236399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.236343 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspcn\" (UniqueName: \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.236783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.236750 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.239668 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.236965 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrjz2\" (UniqueName: \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.248331 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.200419 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.248550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.238344 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.395120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.363698 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspcn\" (UniqueName: \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\") pod \"payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.395120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.364200 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrjz2\" (UniqueName: \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\") pod \"payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.425139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.416446 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:15.425139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.419455 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p"] Oct 30 05:54:15.449350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.449312 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd"] Oct 30 05:54:15.461037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.461002 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88c73b74-f8d7p"] Oct 30 05:54:15.466576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.466548 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1ce8661-896xd"] Oct 30 05:54:15.472140 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.472113 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:15.680230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.680137 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="017cc91c-07b9-48ae-82ae-4e058165e19b" path="/var/lib/kubelet/pods/017cc91c-07b9-48ae-82ae-4e058165e19b/volumes" Oct 30 05:54:15.680893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.680864 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cc78904-1c9f-4eea-8195-d4980a8eef96" path="/var/lib/kubelet/pods/8cc78904-1c9f-4eea-8195-d4980a8eef96/volumes" Oct 30 05:54:15.764988 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:15.764957 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16edc878_2b95_48dc_8aa5_5dbe28a9a2c4.slice/crio-5844f1bdefaa52425e8f5d77f1d0afe3c67c6e6203e396f57a60f6a0fe0855a8 WatchSource:0}: Error finding container 5844f1bdefaa52425e8f5d77f1d0afe3c67c6e6203e396f57a60f6a0fe0855a8: Status 404 returned error can't find the container with id 5844f1bdefaa52425e8f5d77f1d0afe3c67c6e6203e396f57a60f6a0fe0855a8 Oct 30 05:54:15.769347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.769323 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:54:15.803132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:15.803102 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:54:15.804538 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:15.804515 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1db2af24_5eb9_40fb_bdaa_41272710c8fe.slice/crio-fab5a6b6f3866cb0f1e13c11e8077212f9b9579224c54a2d7e755dd525c8c09d WatchSource:0}: Error finding container fab5a6b6f3866cb0f1e13c11e8077212f9b9579224c54a2d7e755dd525c8c09d: Status 404 returned error can't find the container with id fab5a6b6f3866cb0f1e13c11e8077212f9b9579224c54a2d7e755dd525c8c09d Oct 30 05:54:16.686070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.662286 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" event={"ID":"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4","Type":"ContainerStarted","Data":"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe"} Oct 30 05:54:16.686070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.662376 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" event={"ID":"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4","Type":"ContainerStarted","Data":"5844f1bdefaa52425e8f5d77f1d0afe3c67c6e6203e396f57a60f6a0fe0855a8"} Oct 30 05:54:16.686070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.663788 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:16.747463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.735229 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" event={"ID":"1db2af24-5eb9-40fb-bdaa-41272710c8fe","Type":"ContainerStarted","Data":"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783"} Oct 30 05:54:16.747463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.735305 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" event={"ID":"1db2af24-5eb9-40fb-bdaa-41272710c8fe","Type":"ContainerStarted","Data":"fab5a6b6f3866cb0f1e13c11e8077212f9b9579224c54a2d7e755dd525c8c09d"} Oct 30 05:54:16.747463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.736737 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:16.747463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.739203 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" podStartSLOduration=2.739184935 podStartE2EDuration="2.739184935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:16.728128165 +0000 UTC m=+1315.168505355" watchObservedRunningTime="2026-10-30 05:54:16.739184935 +0000 UTC m=+1315.179562129" Oct 30 05:54:16.802346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:16.802293 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" podStartSLOduration=2.802279409 podStartE2EDuration="2.802279409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:16.800192896 +0000 UTC m=+1315.240570089" watchObservedRunningTime="2026-10-30 05:54:16.802279409 +0000 UTC m=+1315.242656659" Oct 30 05:54:17.048443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:17.048242 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d"] Oct 30 05:54:17.063228 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:17.062826 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43f71aef-n4w8d"] Oct 30 05:54:17.816863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:17.816821 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe0a44e6-ee45-4623-be40-4b90a9eab753" path="/var/lib/kubelet/pods/fe0a44e6-ee45-4623-be40-4b90a9eab753/volumes" Oct 30 05:54:22.164530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.164500 6171 scope.go:122] "RemoveContainer" containerID="5eb89c0a562c1bb1f8a0fa57e6d738246804e9cbddfc28eb612fa8a9a9376b38" Oct 30 05:54:22.191547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.191512 6171 scope.go:122] "RemoveContainer" containerID="71b9a77a806ae2d22faf7558bb7a0a5f967205e962a11edfbffad47120bc141f" Oct 30 05:54:22.203685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.203662 6171 scope.go:122] "RemoveContainer" containerID="832fabdf50f69afbd9c2cecc26adca4eed3ebaf3eec2058b1511e644bdb14b3b" Oct 30 05:54:22.216547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.216526 6171 scope.go:122] "RemoveContainer" containerID="0e9d15d111f7befff0a7ab66cbb37af4878f48d15e3ae791749a846b68b5955a" Oct 30 05:54:22.229214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.229190 6171 scope.go:122] "RemoveContainer" containerID="b174338d459b16d246019f87f7c18d8c87e2700bb0fe0e1fd43bc01cfcc5d392" Oct 30 05:54:22.241568 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.241542 6171 scope.go:122] "RemoveContainer" containerID="ec2fbff575b272983824f00133c5744a6703b7f2ad3566b62034cf5664775157" Oct 30 05:54:22.253801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.253778 6171 scope.go:122] "RemoveContainer" containerID="22017ca6aebaffda1e9dafc352eea022adc4524394c5a3896cb8fb225c99628a" Oct 30 05:54:22.265127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.265103 6171 scope.go:122] "RemoveContainer" containerID="be093b6f847802eb6335d03e3c303b0b16a81900533e911330f86e1812c9e642" Oct 30 05:54:22.279715 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:22.279691 6171 scope.go:122] "RemoveContainer" containerID="18c00e0ff403506f1c8567c17677b818edf0cc0a3669e8048f7cfe6159cede68" Oct 30 05:54:25.010378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.010318 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.099315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.099273 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.242551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.242146 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.435289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.433392 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:54:25.497645 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.481960 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.576709 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.576462 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.700156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.697033 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.721789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:25.721747 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.014069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.011930 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:54:26.086760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.086710 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 05:54:26.124332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.124286 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.155114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.126861 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.159659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.159562 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded" Oct 30 05:54:26.217658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.217626 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:54:26.217955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.217941 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:54:26.243190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.243148 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.243835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.243812 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:54:26.583974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.564660 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.583974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.580157 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.583974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.581471 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:54:26.583974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.581535 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:54:26.816078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.760311 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 05:54:26.816078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:26.811184 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:54:27.024888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.019886 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.139071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.133246 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:54:27.149162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.149121 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.150728 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.150705 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:54:27.190073 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:27.186962 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:54:27.240809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.220490 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:27.240809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.220567 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.272073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.245890 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.328317 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.327730 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:27.328317 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.327798 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.504244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.504197 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.572927 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.572139 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 05:54:27.600298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.598257 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 05:54:27.739066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.735607 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 05:54:27.764298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.761123 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.930250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.926072 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:27.930250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:27.926154 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:28.185678 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.183966 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:54:28.207707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.200243 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:28.218285 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.200291 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:54:28.255781 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.255734 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:54:28.463724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.463635 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 05:54:28.544072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.539338 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 05:54:28.562537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.562511 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:54:28.573837 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.573805 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:28.575107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.575082 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 05:54:28.633181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.597619 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 05:54:28.686520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.686496 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 05:54:28.905083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.903798 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:54:28.963975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:28.963502 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:54:32.039768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:32.039726 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz"] Oct 30 05:54:32.051416 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:32.051387 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-168ced75-rrqtz"] Oct 30 05:54:33.676979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:33.676944 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6562722b-3a60-4fd5-89cf-c6a18dd92494" path="/var/lib/kubelet/pods/6562722b-3a60-4fd5-89cf-c6a18dd92494/volumes" Oct 30 05:54:34.375256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:34.375212 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:34.375477 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:34.375287 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:37.015968 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.015912 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.045144 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.045901 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.072851 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1b2243ef\"/\"openshift-service-ca.crt\"" Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.072890 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-1b2243ef\"/\"kube-root-ca.crt\"" Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.073421 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1b2243ef\"/\"default-dockercfg-nv9l2\"" Oct 30 05:54:37.204378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.073443 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-1b2243ef\"/\"e2e-lifecycle-model-1b2243ef-kserve-self-signed-certs\"" Oct 30 05:54:37.211499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.211441 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bk84d\" (UniqueName: \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.211695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.211673 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.211950 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.211927 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.212114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.212088 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.212166 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.212132 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.212278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.212255 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.315595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.315500 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.315834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.315817 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.318572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.318543 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.319917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.319889 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bk84d\" (UniqueName: \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.324766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.324740 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.328071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.325139 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.328742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.319108 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.329237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.329215 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.330857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.330835 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.391694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.390389 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.415877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.414480 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.433827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.432995 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bk84d\" (UniqueName: \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\") pod \"e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:37.682556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:37.682444 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:38.081774 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:38.033989 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c248106_d4bc_4ede_ba9f_552da4945b3c.slice/crio-db227eaaed37595d1b833931f26863fa81a68e90f93d73b358f8800122ea262d WatchSource:0}: Error finding container db227eaaed37595d1b833931f26863fa81a68e90f93d73b358f8800122ea262d: Status 404 returned error can't find the container with id db227eaaed37595d1b833931f26863fa81a68e90f93d73b358f8800122ea262d Oct 30 05:54:38.081774 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.038284 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:54:38.736290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.736257 6171 generic.go:356] "Generic (PLEG): container finished" podID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerID="11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f" exitCode=0 Oct 30 05:54:38.736290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.736287 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerDied","Data":"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f"} Oct 30 05:54:38.738550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.738524 6171 scope.go:122] "RemoveContainer" containerID="11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f" Oct 30 05:54:38.738550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.738547 6171 generic.go:356] "Generic (PLEG): container finished" podID="961d6e36-ecec-4957-abdc-76ea72000259" containerID="185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008" exitCode=0 Oct 30 05:54:38.738760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.738614 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerDied","Data":"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008"} Oct 30 05:54:38.739029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.739007 6171 scope.go:122] "RemoveContainer" containerID="185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008" Oct 30 05:54:38.740778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.740753 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.740848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.740830 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.740921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.740889 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.741697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.741574 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" event={"ID":"5c248106-d4bc-4ede-ba9f-552da4945b3c","Type":"ContainerStarted","Data":"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b"} Oct 30 05:54:38.741697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.741602 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" event={"ID":"5c248106-d4bc-4ede-ba9f-552da4945b3c","Type":"ContainerStarted","Data":"db227eaaed37595d1b833931f26863fa81a68e90f93d73b358f8800122ea262d"} Oct 30 05:54:38.741835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.741698 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.741835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.741758 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.741835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.741792 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:38.893324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:38.893270 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" podStartSLOduration=2.893252767 podStartE2EDuration="2.893252767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:38.886710024 +0000 UTC m=+1337.327087221" watchObservedRunningTime="2026-10-30 05:54:38.893252767 +0000 UTC m=+1337.333629961" Oct 30 05:54:39.133125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.132974 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js"] Oct 30 05:54:39.179105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.169609 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eef45f53-984js"] Oct 30 05:54:39.813069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.803432 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cad0fa-599f-4bdd-a41c-c4c93c97b83d" path="/var/lib/kubelet/pods/21cad0fa-599f-4bdd-a41c-c4c93c97b83d/volumes" Oct 30 05:54:39.822642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.822609 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:39.903176 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.903136 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerStarted","Data":"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4"} Oct 30 05:54:39.949079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:39.945768 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerStarted","Data":"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc"} Oct 30 05:54:40.035403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.034562 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:40.035403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.034625 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:40.035403 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.035018 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:54:40.562084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.557302 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:54:40.562084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.557436 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:40.562084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:40.557484 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:41.038065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.032088 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:41.044162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.032147 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:41.475968 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.474220 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:41.651088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.648480 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:41.651088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.648563 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:41.771839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:41.771737 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:42.039125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.038120 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:42.039125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.038280 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:42.459016 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.458930 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:54:42.459373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.459355 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 05:54:42.474652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.474626 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 05:54:42.490108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.475021 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.513772 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.513887 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.513945 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.514030 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.529528 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.529797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.530017 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.530206 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.542069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.530902 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5tcc\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.556069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.544285 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-f22a52-openshift-default-dockercfg-fmtw2\"" Oct 30 05:54:42.583077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.558339 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:42.583077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.582915 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:42.659073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.651738 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:42.659073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.653627 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:42.673306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.673269 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.699072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.673647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.699439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.699406 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.699619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.699604 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.699750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.699737 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.700078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.700036 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5tcc\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.700331 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.700312 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.700535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.700519 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.700659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.700641 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.715112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.713595 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.715112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.682736 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.715112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.684208 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.743647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.735368 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.743647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.735535 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.743647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.735997 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.775069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.774969 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:42.799065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.780246 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:54:42.799065 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.794316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.824066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.822824 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5tcc\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.852780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.841294 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\") pod \"e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:42.860782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.860755 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 05:54:42.944071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:42.935641 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:43.060409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.059936 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:43.060409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.059995 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:43.469186 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:43.469148 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce686bc_222c_4f3d_bfae_1862a05db9b7.slice/crio-04b41a8803918073c126e423f1b343d2759d52393920a916f1408045550bbef8 WatchSource:0}: Error finding container 04b41a8803918073c126e423f1b343d2759d52393920a916f1408045550bbef8: Status 404 returned error can't find the container with id 04b41a8803918073c126e423f1b343d2759d52393920a916f1408045550bbef8 Oct 30 05:54:43.472245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.472209 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:43.472358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.472285 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:43.472358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.472312 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:43.473919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.473893 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 05:54:43.553567 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.553489 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:43.553694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:43.553588 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:44.041071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.040188 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:44.041071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.040248 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:44.554245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.554155 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:44.554245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.554222 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:44.558881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.558838 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerStarted","Data":"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d"} Oct 30 05:54:44.559102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.558899 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerStarted","Data":"04b41a8803918073c126e423f1b343d2759d52393920a916f1408045550bbef8"} Oct 30 05:54:44.689071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.685987 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podStartSLOduration=3.685967111 podStartE2EDuration="3.685967111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:44.668491375 +0000 UTC m=+1343.108868568" watchObservedRunningTime="2026-10-30 05:54:44.685967111 +0000 UTC m=+1343.126344307" Oct 30 05:54:44.979068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:44.976467 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:45.043082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.041097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:45.044061 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.044016 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:45.044279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.044251 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:45.553427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.553393 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:45.553597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.553463 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:45.569345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.569318 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:45.570394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:45.570373 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:54:46.130086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:46.097990 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:46.140421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:46.140346 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:46.559013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:46.558967 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:46.559327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:46.559296 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:47.073670 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.073633 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:47.106870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.106827 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:47.559084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.558454 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 05:54:47.559084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.558512 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 05:54:47.774199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.774169 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:47.824256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:47.824179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:48.032742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.032703 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:54:48.032946 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.032765 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.165:15021/healthz/ready\": dial tcp 10.133.0.165:15021: connect: connection refused" Oct 30 05:54:48.178128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.169510 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s"] Oct 30 05:54:48.196072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.183250 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3872f64-wjb5s"] Oct 30 05:54:48.337064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.336745 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:54:48.561090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.561019 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:54:48.695122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.695077 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:54:48.791708 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.791678 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:54:48.994815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:48.994778 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 05:54:49.001515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.001492 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.038341 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.038315 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4fc75f-openshift-default-dockercfg-6gtb9\"" Oct 30 05:54:49.042679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.042658 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:54:49.044018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.043994 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 05:54:49.044976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.044956 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 05:54:49.055092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055011 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055210 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055103 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72jzf\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055331 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055545 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055525 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055604 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055705 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055760 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.055804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.055789 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181030 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181128 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72jzf\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181463 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181582 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181629 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181677 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.181708 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.187092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.184443 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.188892 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.188866 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.222988 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.201240 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.255128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.247227 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.336068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.333389 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.345893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.345812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.349631 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.349599 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72jzf\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.366307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.366210 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.367970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.367912 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.716131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.650789 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:49.965070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.961719 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77e4d840-8240-45e1-8daf-b70feb18e4b1" path="/var/lib/kubelet/pods/77e4d840-8240-45e1-8daf-b70feb18e4b1/volumes" Oct 30 05:54:49.965070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.963621 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 05:54:49.996700 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.995651 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 05:54:50.015279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:49.997673 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.048368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.044613 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-f22a52\"" Oct 30 05:54:50.048368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.045408 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-f22a52-dockercfg-pxgr6\"" Oct 30 05:54:50.210087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.207380 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.210087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.207655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glgh2\" (UniqueName: \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.338134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.337469 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.338134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.337667 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glgh2\" (UniqueName: \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.347641 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.344772 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.440070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.420712 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glgh2\" (UniqueName: \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\") pod \"payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:50.560003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:50.559972 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:51.113495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.110919 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:51.193627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.193576 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:51.196729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.196682 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:54:51.216074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.214641 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 05:54:51.220550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.218807 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerStarted","Data":"c33e646e5c78fa722431716e3dd7f94b1e8cbb099dff8c594808001cb9409cfc"} Oct 30 05:54:51.391702 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:51.391651 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 05:54:51.392117 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:51.392031 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c7e4b6_b102_4e58_9ab8_04df57b6b1df.slice/crio-6b0fcf0dca0340ccb7984e909798dd9a802466f06f3d26a5d022cc9831f15239 WatchSource:0}: Error finding container 6b0fcf0dca0340ccb7984e909798dd9a802466f06f3d26a5d022cc9831f15239: Status 404 returned error can't find the container with id 6b0fcf0dca0340ccb7984e909798dd9a802466f06f3d26a5d022cc9831f15239 Oct 30 05:54:52.046606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.046572 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s"] Oct 30 05:54:52.059424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.059391 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-05036bc6-gv94s"] Oct 30 05:54:52.066077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.066037 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx"] Oct 30 05:54:52.072420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.072388 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c69d5760-jkfcx"] Oct 30 05:54:52.235849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.235794 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" event={"ID":"00c7e4b6-b102-4e58-9ab8-04df57b6b1df","Type":"ContainerStarted","Data":"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa"} Oct 30 05:54:52.236411 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.235868 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" event={"ID":"00c7e4b6-b102-4e58-9ab8-04df57b6b1df","Type":"ContainerStarted","Data":"6b0fcf0dca0340ccb7984e909798dd9a802466f06f3d26a5d022cc9831f15239"} Oct 30 05:54:52.236533 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.236494 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:54:52.238419 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.238355 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerStarted","Data":"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a"} Oct 30 05:54:52.281314 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.281266 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" podStartSLOduration=3.281252875 podStartE2EDuration="3.281252875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:52.278102762 +0000 UTC m=+1350.718479956" watchObservedRunningTime="2026-10-30 05:54:52.281252875 +0000 UTC m=+1350.721630068" Oct 30 05:54:52.335336 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.335234 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podStartSLOduration=4.335217954 podStartE2EDuration="4.335217954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:52.328827782 +0000 UTC m=+1350.769204974" watchObservedRunningTime="2026-10-30 05:54:52.335217954 +0000 UTC m=+1350.775595146" Oct 30 05:54:52.388603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.388572 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 05:54:52.394513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.394488 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.413347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.413319 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-cc33d3-dockercfg-twp56\"" Oct 30 05:54:52.413483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.413367 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-cc33d3\"" Oct 30 05:54:52.414392 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.414371 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 05:54:52.533134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.532259 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.533134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.532728 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjhsl\" (UniqueName: \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.630202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.627108 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 05:54:52.665071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.661001 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.665071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.661502 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjhsl\" (UniqueName: \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.665071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.662776 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.700108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.698179 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:52.700108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.699704 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:52.753852 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.753784 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjhsl\" (UniqueName: \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\") pod \"payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:52.892898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.889687 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:52.895112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.893971 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:52.895112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.894198 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vm8jq\" (UniqueName: \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:52.969137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:52.969100 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 05:54:53.035072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.029256 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:53.035072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.029476 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vm8jq\" (UniqueName: \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:53.035072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.031668 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:53.049067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.041811 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:53.116116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.111026 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vm8jq\" (UniqueName: \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\") pod \"payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:53.244650 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.244573 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:53.272637 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.272607 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:53.278119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.278027 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:54:53.431121 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:53.431039 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4f5b3e_8631_4de5_a254_29e55f219a6b.slice/crio-7a77e53137803202d8ebbef4f8f3663f1b46706752c5a57c75857e5ba0ad50b3 WatchSource:0}: Error finding container 7a77e53137803202d8ebbef4f8f3663f1b46706752c5a57c75857e5ba0ad50b3: Status 404 returned error can't find the container with id 7a77e53137803202d8ebbef4f8f3663f1b46706752c5a57c75857e5ba0ad50b3 Oct 30 05:54:53.432092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.432001 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 05:54:53.590713 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:53.590682 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7322401_c8ba_474a_817a_0558e3b62a1e.slice/crio-301b3f57ddd2cd254399d6ae9d6b989a7636e8b1dd6faede07553421b7230fbc WatchSource:0}: Error finding container 301b3f57ddd2cd254399d6ae9d6b989a7636e8b1dd6faede07553421b7230fbc: Status 404 returned error can't find the container with id 301b3f57ddd2cd254399d6ae9d6b989a7636e8b1dd6faede07553421b7230fbc Oct 30 05:54:53.592548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.592468 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 05:54:53.678318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.678261 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57c8275-0087-4666-b7fa-b1a8785040e9" path="/var/lib/kubelet/pods/b57c8275-0087-4666-b7fa-b1a8785040e9/volumes" Oct 30 05:54:53.678912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:53.678892 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5" path="/var/lib/kubelet/pods/ff4a50f0-158c-4c58-bd8e-19eaa1a94ac5/volumes" Oct 30 05:54:54.357069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.355141 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" event={"ID":"ec4f5b3e-8631-4de5-a254-29e55f219a6b","Type":"ContainerStarted","Data":"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9"} Oct 30 05:54:54.357069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.355217 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" event={"ID":"ec4f5b3e-8631-4de5-a254-29e55f219a6b","Type":"ContainerStarted","Data":"7a77e53137803202d8ebbef4f8f3663f1b46706752c5a57c75857e5ba0ad50b3"} Oct 30 05:54:54.357069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.356751 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:54:54.372070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.361682 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" event={"ID":"d7322401-c8ba-474a-817a-0558e3b62a1e","Type":"ContainerStarted","Data":"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56"} Oct 30 05:54:54.372070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.361736 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" event={"ID":"d7322401-c8ba-474a-817a-0558e3b62a1e","Type":"ContainerStarted","Data":"301b3f57ddd2cd254399d6ae9d6b989a7636e8b1dd6faede07553421b7230fbc"} Oct 30 05:54:54.372070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.361767 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:54:54.421863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.421794 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" podStartSLOduration=2.421770961 podStartE2EDuration="2.421770961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:54.409103086 +0000 UTC m=+1352.849480281" watchObservedRunningTime="2026-10-30 05:54:54.421770961 +0000 UTC m=+1352.862148156" Oct 30 05:54:54.471110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:54.463609 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" podStartSLOduration=2.463590627 podStartE2EDuration="2.463590627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:54.461385445 +0000 UTC m=+1352.901762640" watchObservedRunningTime="2026-10-30 05:54:54.463590627 +0000 UTC m=+1352.903967820" Oct 30 05:54:56.383182 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:54:56.383143 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:54:56.728961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.728931 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:54:56.735515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.735494 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:56.754569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.754542 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:54:56.758265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.758233 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4fc75f\"" Oct 30 05:54:56.758386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.758276 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4fc75f-dockercfg-vrrpt\"" Oct 30 05:54:56.816069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.810567 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:56.816069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.810785 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q88p8\" (UniqueName: \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:56.917072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.916469 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q88p8\" (UniqueName: \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:56.917072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.916857 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:56.945070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:56.917869 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:57.340005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:57.339227 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q88p8\" (UniqueName: \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\") pod \"payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:57.354497 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:57.354466 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:57.674180 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:54:57.674144 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76870fdd_4e06_4fcd_ac32_2c9689140113.slice/crio-08a21dd42b96601ca907fcb42088da0c53460781d5e64684e12a44e62f314231 WatchSource:0}: Error finding container 08a21dd42b96601ca907fcb42088da0c53460781d5e64684e12a44e62f314231: Status 404 returned error can't find the container with id 08a21dd42b96601ca907fcb42088da0c53460781d5e64684e12a44e62f314231 Oct 30 05:54:57.679122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:57.679090 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:54:58.409836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:58.409793 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" event={"ID":"76870fdd-4e06-4fcd-ac32-2c9689140113","Type":"ContainerStarted","Data":"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118"} Oct 30 05:54:58.410083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:58.409892 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" event={"ID":"76870fdd-4e06-4fcd-ac32-2c9689140113","Type":"ContainerStarted","Data":"08a21dd42b96601ca907fcb42088da0c53460781d5e64684e12a44e62f314231"} Oct 30 05:54:58.410083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:58.409920 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:54:58.452575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:58.452524 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" podStartSLOduration=2.45250956 podStartE2EDuration="2.45250956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:58.449990999 +0000 UTC m=+1356.890368188" watchObservedRunningTime="2026-10-30 05:54:58.45250956 +0000 UTC m=+1356.892886752" Oct 30 05:54:59.892913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:54:59.892878 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:55:03.041548 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:03.041467 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf"] Oct 30 05:55:03.052039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:03.052007 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-022c89b8-dn9jf"] Oct 30 05:55:03.308354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:03.308259 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 05:55:03.675652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:03.675563 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccf2ed18-fd10-409d-9e6a-375fe2a373f5" path="/var/lib/kubelet/pods/ccf2ed18-fd10-409d-9e6a-375fe2a373f5/volumes" Oct 30 05:55:05.374470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:05.374439 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 05:55:06.035316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:06.035283 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp"] Oct 30 05:55:06.046978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:06.046947 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-763e519d-dmntp"] Oct 30 05:55:06.427345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:06.427246 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 05:55:07.706083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:07.695385 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a639fda0-ccc6-4e1d-b653-400094e86ea8" path="/var/lib/kubelet/pods/a639fda0-ccc6-4e1d-b653-400094e86ea8/volumes" Oct 30 05:55:09.423141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:09.423105 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:55:10.058623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:10.058595 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54"] Oct 30 05:55:10.065234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:10.065208 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2b915f7f-vnj54"] Oct 30 05:55:11.764434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:11.764385 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="adaadbad-7a54-49cb-9929-58c80d656db3" path="/var/lib/kubelet/pods/adaadbad-7a54-49cb-9929-58c80d656db3/volumes" Oct 30 05:55:12.035964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:12.035891 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl"] Oct 30 05:55:12.052194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:12.052164 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1b14588b-f69sl"] Oct 30 05:55:12.493834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:12.492168 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:55:12.493834 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:12.492236 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:13.798254 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:13.798214 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="448e29c0-d09d-4dde-9220-e5b04d1c1707" path="/var/lib/kubelet/pods/448e29c0-d09d-4dde-9220-e5b04d1c1707/volumes" Oct 30 05:55:16.519098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.519068 6171 generic.go:356] "Generic (PLEG): container finished" podID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerID="e91641a98e117442b4a47c23e4fe85a27662f968c13f9acffb125b310691be1b" exitCode=0 Oct 30 05:55:16.519513 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.519131 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerDied","Data":"e91641a98e117442b4a47c23e4fe85a27662f968c13f9acffb125b310691be1b"} Oct 30 05:55:16.519600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.519586 6171 scope.go:122] "RemoveContainer" containerID="e91641a98e117442b4a47c23e4fe85a27662f968c13f9acffb125b310691be1b" Oct 30 05:55:16.521536 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.521508 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:16.521624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.521579 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:16.521624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:16.521610 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:17.259303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.259249 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:55:17.265328 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.265301 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.288867 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.288827 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-b202a0\"" Oct 30 05:55:17.289582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.289561 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-b202a0-dockercfg-vnswb\"" Oct 30 05:55:17.306089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.304355 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:55:17.464073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.462639 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.464073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.462752 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f85w8\" (UniqueName: \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.702152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.646622 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.770120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.758411 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f85w8\" (UniqueName: \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.770120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.654034 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.845069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.838982 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f85w8\" (UniqueName: \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\") pod \"payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.930632 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.919528 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:17.996129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:17.976345 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerStarted","Data":"19da64d8eb1f7d43928796804247eb1373d4f79dfc298303e2a1ce32194cb919"} Oct 30 05:55:18.397660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.397575 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:55:18.432189 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.432148 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:18.432573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.432524 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:18.661153 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:55:18.661119 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b964009_2b1c_4739_9ea3_3205beb9e37e.slice/crio-72b5bddc9b8610d59734bc57ccf1754a625171a8032c45f9a25821df4a8cc1eb WatchSource:0}: Error finding container 72b5bddc9b8610d59734bc57ccf1754a625171a8032c45f9a25821df4a8cc1eb: Status 404 returned error can't find the container with id 72b5bddc9b8610d59734bc57ccf1754a625171a8032c45f9a25821df4a8cc1eb Oct 30 05:55:18.664387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.664359 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:55:18.979331 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.979299 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" event={"ID":"3b964009-2b1c-4739-9ea3-3205beb9e37e","Type":"ContainerStarted","Data":"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4"} Oct 30 05:55:18.979735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.979360 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" event={"ID":"3b964009-2b1c-4739-9ea3-3205beb9e37e","Type":"ContainerStarted","Data":"72b5bddc9b8610d59734bc57ccf1754a625171a8032c45f9a25821df4a8cc1eb"} Oct 30 05:55:18.979735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:18.979390 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:19.041949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:19.041901 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" podStartSLOduration=2.041888058 podStartE2EDuration="2.041888058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:19.034743114 +0000 UTC m=+1377.475120307" watchObservedRunningTime="2026-10-30 05:55:19.041888058 +0000 UTC m=+1377.482265306" Oct 30 05:55:19.304122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:19.303024 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:19.304122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:19.303114 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:20.311634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:20.311596 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:20.312195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:20.311660 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:21.295075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.291536 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:21.295075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.291601 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:21.620304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.620223 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:55:21.634213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.634184 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.637898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.637867 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:55:21.652797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.652774 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a3efab09\"" Oct 30 05:55:21.664494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.664442 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a3efab09-dockercfg-4rv2b\"" Oct 30 05:55:21.680182 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.680153 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.680693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.680668 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kct6l\" (UniqueName: \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.685888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.685867 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:55:21.693079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.693059 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:21.794131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.793280 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4rpv\" (UniqueName: \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:21.794131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.793434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.794131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.793587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:21.794131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.793803 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kct6l\" (UniqueName: \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.813122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.809275 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a3efab09\"" Oct 30 05:55:21.829669 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.827979 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:21.968118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.954676 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:21.968118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:21.954962 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4rpv\" (UniqueName: \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:22.148079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.131493 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:22.332086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.327883 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:22.332086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.327955 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:22.529811 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.520723 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:55:22.604364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.589809 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kct6l\" (UniqueName: \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\") pod \"payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:22.604364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.595701 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4rpv\" (UniqueName: \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\") pod \"payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:22.681131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.666659 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj"] Oct 30 05:55:22.708628 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.708583 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.177:8000/ready\": context deadline exceeded" Oct 30 05:55:22.812245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.803154 6171 scope.go:122] "RemoveContainer" containerID="caf3f41c32faf1a9a2f3f41fb25a95e9d38a9752e29ddc658bde3507669d174b" Oct 30 05:55:22.812245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.808684 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:22.891777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:22.891620 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:23.153457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.153416 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.204909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.180632 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.204909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.194624 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 05:55:23.362616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.362581 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:23.363935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.363901 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:23.392071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.389782 6171 scope.go:122] "RemoveContainer" containerID="9f4eab8b4245e508643baed56d9f98b3ef6272c94135837ab65830d5f253def6" Oct 30 05:55:23.565256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.556001 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9b23add5-ms2rj"] Oct 30 05:55:23.565256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.558948 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 05:55:23.637414 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.637360 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.680581 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.640167 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.718327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.684647 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.719116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.685013 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 05:55:23.719450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.717635 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.177:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:55:23.720235 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.720212 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:55:23.905348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.905310 6171 scope.go:122] "RemoveContainer" containerID="5f45cae5fa86384d1ddae35722713080f2946d003c3ef10b00abe0cf8a341b03" Oct 30 05:55:23.935556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.935516 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d7095c-7690-4c21-8657-f107d417f34d" path="/var/lib/kubelet/pods/78d7095c-7690-4c21-8657-f107d417f34d/volumes" Oct 30 05:55:23.936296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.936281 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:55:23.949114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.949094 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 05:55:23.958421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.957598 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:23.965940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.965870 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 05:55:23.973938 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.973867 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-862dc2-openshift-default-dockercfg-8mcqs\"" Oct 30 05:55:23.980266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.980238 6171 scope.go:122] "RemoveContainer" containerID="f6f85c102e8b57b0ea8c81c29cd031425cf319e74230f6709c25d08fca3c9557" Oct 30 05:55:23.980383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.980368 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66"] Oct 30 05:55:23.992797 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:23.992766 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3805d87-9vc66"] Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021262 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqrtc\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021324 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021419 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021564 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021605 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021680 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021741 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021789 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.022493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.021838 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.052105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.051580 6171 scope.go:122] "RemoveContainer" containerID="a16a349425efb2cbc6cc2e4ade0fabb8375e6d9bcd4bde80f12f2dfae113b7f3" Oct 30 05:55:24.089749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.087312 6171 scope.go:122] "RemoveContainer" containerID="8865d06e575fa5d4a529dd5da78ce95bbf94321dcc91875a4969d6e4e50d41e1" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129299 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129493 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129628 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129695 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129742 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.129806 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.137137 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqrtc\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.137209 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.137944 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.138401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.138642 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.138762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.138879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.138815 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.142333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.142288 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.142474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.142449 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.162371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.162345 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqrtc\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.162515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.162369 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\") pod \"e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.175196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.175175 6171 scope.go:122] "RemoveContainer" containerID="5fe68aa8699dd2901a0972fe44e8fcb20eac7d9bc33ada0f5a6c97ce5fd1c81e" Oct 30 05:55:24.319585 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.319552 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 05:55:24.320064 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.319650 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 05:55:24.322338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.322317 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:24.474203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.474176 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:55:24.485340 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:55:24.485295 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d719668_a732_47c3_80ac_98525d78bd2f.slice/crio-79d93ef9c83a9f87af9f50402b0a7bdab68aaf23e63f81338cea6f7621c3c478 WatchSource:0}: Error finding container 79d93ef9c83a9f87af9f50402b0a7bdab68aaf23e63f81338cea6f7621c3c478: Status 404 returned error can't find the container with id 79d93ef9c83a9f87af9f50402b0a7bdab68aaf23e63f81338cea6f7621c3c478 Oct 30 05:55:24.587396 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:55:24.570397 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e158c7c_945b_4a2a_b184_ab2c2500bc42.slice/crio-a777faca44d92935ce3b02f351b0285a490bc6612c9886b0624411f0101d8717 WatchSource:0}: Error finding container a777faca44d92935ce3b02f351b0285a490bc6612c9886b0624411f0101d8717: Status 404 returned error can't find the container with id a777faca44d92935ce3b02f351b0285a490bc6612c9886b0624411f0101d8717 Oct 30 05:55:24.717692 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.717409 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:55:24.733135 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.733086 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 05:55:24.766469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.766213 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:24.766469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.766279 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:24.766469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.766310 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:55:24.950072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.949702 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerStarted","Data":"9305a9e20ea8bc0efed20121f782dcadf74b0485f4b769569c594315fb3717e4"} Oct 30 05:55:24.950072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.949770 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerStarted","Data":"d593d4ff0803a2dba8a1071839fc454f42aadb1409d1ad27ee380ed40bde9b85"} Oct 30 05:55:24.951980 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.951949 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" event={"ID":"7d719668-a732-47c3-80ac-98525d78bd2f","Type":"ContainerStarted","Data":"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec"} Oct 30 05:55:24.952120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.952005 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" event={"ID":"7d719668-a732-47c3-80ac-98525d78bd2f","Type":"ContainerStarted","Data":"79d93ef9c83a9f87af9f50402b0a7bdab68aaf23e63f81338cea6f7621c3c478"} Oct 30 05:55:24.952872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.952851 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:24.955165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.955139 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" event={"ID":"3e158c7c-945b-4a2a-b184-ab2c2500bc42","Type":"ContainerStarted","Data":"c9e2d2bddc5e3c805de29c60d85f4a429bca3225ca272f50a5b751b87558aa40"} Oct 30 05:55:24.955298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.955189 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" event={"ID":"3e158c7c-945b-4a2a-b184-ab2c2500bc42","Type":"ContainerStarted","Data":"a777faca44d92935ce3b02f351b0285a490bc6612c9886b0624411f0101d8717"} Oct 30 05:55:24.955298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:24.955267 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:25.007377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.006197 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podStartSLOduration=3.006181115 podStartE2EDuration="3.006181115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:24.996026679 +0000 UTC m=+1383.436403872" watchObservedRunningTime="2026-10-30 05:55:25.006181115 +0000 UTC m=+1383.446558309" Oct 30 05:55:25.045066 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.044192 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" podStartSLOduration=4.0441725 podStartE2EDuration="4.0441725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:25.037990002 +0000 UTC m=+1383.478367197" watchObservedRunningTime="2026-10-30 05:55:25.0441725 +0000 UTC m=+1383.484549694" Oct 30 05:55:25.082492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.082432 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" podStartSLOduration=4.082413449 podStartE2EDuration="4.082413449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:25.077801718 +0000 UTC m=+1383.518178913" watchObservedRunningTime="2026-10-30 05:55:25.082413449 +0000 UTC m=+1383.522790642" Oct 30 05:55:25.327883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.326737 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:25.331071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.328447 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:55:25.331071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.328508 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:55:25.331071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.329648 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:55:25.338758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.331574 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:55:25.338758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.333023 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:55:25.677221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:25.677143 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93317926-5246-42e4-b051-0fa3e16d0827" path="/var/lib/kubelet/pods/93317926-5246-42e4-b051-0fa3e16d0827/volumes" Oct 30 05:55:26.038748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:26.038717 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn"] Oct 30 05:55:26.045034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:26.045009 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d689555-rtgqn"] Oct 30 05:55:26.408133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:26.407469 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:26.552592 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:55:26.552549 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:55:26.977602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:26.977568 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:26.978369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:26.978348 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:55:27.772069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:27.767933 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00725e0e-3a03-4e20-8aa3-50c733c5159d" path="/var/lib/kubelet/pods/00725e0e-3a03-4e20-8aa3-50c733c5159d/volumes" Oct 30 05:55:30.220467 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:30.220439 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:55:35.033181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.033148 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf"] Oct 30 05:55:35.045240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.045209 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm"] Oct 30 05:55:35.051268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.051238 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-133fe481-sktzm"] Oct 30 05:55:35.056493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.056473 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6be5590e-kkcjf"] Oct 30 05:55:35.679111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.679072 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05263769-0ea2-4786-ad5b-7f3d56bb9f25" path="/var/lib/kubelet/pods/05263769-0ea2-4786-ad5b-7f3d56bb9f25/volumes" Oct 30 05:55:35.679694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.679675 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="251c399a-4fb5-4247-b277-c716229a5524" path="/var/lib/kubelet/pods/251c399a-4fb5-4247-b277-c716229a5524/volumes" Oct 30 05:55:35.969918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:35.969841 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:55:36.980388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:36.980360 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:55:38.033870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:38.033832 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd"] Oct 30 05:55:38.044576 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:38.044545 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c9dbc3-s46qd"] Oct 30 05:55:39.676157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:39.676125 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23eeeb8f-54a1-4419-a4af-ac460e6de715" path="/var/lib/kubelet/pods/23eeeb8f-54a1-4419-a4af-ac460e6de715/volumes" Oct 30 05:55:48.508730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.508700 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9201-1a6e9d"] Oct 30 05:55:48.514134 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.514114 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:48.565073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.544372 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9201-1a6e9d"] Oct 30 05:55:48.790898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.788249 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wtw\" (UniqueName: \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\") pod \"test-curl-9201-1a6e9d\" (UID: \"0dc8e44e-7877-496b-9a4b-8a171c3de7a2\") " pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:48.791636 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.791588 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4m56v\"" Oct 30 05:55:48.889941 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.889902 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wtw\" (UniqueName: \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\") pod \"test-curl-9201-1a6e9d\" (UID: \"0dc8e44e-7877-496b-9a4b-8a171c3de7a2\") " pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:48.913334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:48.913295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wtw\" (UniqueName: \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\") pod \"test-curl-9201-1a6e9d\" (UID: \"0dc8e44e-7877-496b-9a4b-8a171c3de7a2\") " pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:49.085154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:49.085033 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:49.509006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:49.508976 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9201-1a6e9d"] Oct 30 05:55:49.510601 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:55:49.510569 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dc8e44e_7877_496b_9a4b_8a171c3de7a2.slice/crio-d386c7641bd972b4110854ee835955a5d1d5ea29cca5a97125cdac60e139ca50 WatchSource:0}: Error finding container d386c7641bd972b4110854ee835955a5d1d5ea29cca5a97125cdac60e139ca50: Status 404 returned error can't find the container with id d386c7641bd972b4110854ee835955a5d1d5ea29cca5a97125cdac60e139ca50 Oct 30 05:55:50.055809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:50.055761 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9201-1a6e9d" event={"ID":"0dc8e44e-7877-496b-9a4b-8a171c3de7a2","Type":"ContainerStarted","Data":"c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6"} Oct 30 05:55:50.055995 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:50.055826 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9201-1a6e9d" event={"ID":"0dc8e44e-7877-496b-9a4b-8a171c3de7a2","Type":"ContainerStarted","Data":"d386c7641bd972b4110854ee835955a5d1d5ea29cca5a97125cdac60e139ca50"} Oct 30 05:55:50.145901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:50.145807 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9201-1a6e9d" podStartSLOduration=2.145783635 podStartE2EDuration="2.145783635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:50.14024914 +0000 UTC m=+1408.580626334" watchObservedRunningTime="2026-10-30 05:55:50.145783635 +0000 UTC m=+1408.586160830" Oct 30 05:55:51.056671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.056642 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9201-1a6e9d"] Oct 30 05:55:51.067098 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.066302 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9201-1a6e9d"] Oct 30 05:55:51.675496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.675465 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz"] Oct 30 05:55:51.676458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.676425 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" containerName="test-curl-9201-1a6e9d" Oct 30 05:55:51.676458 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.676445 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" containerName="test-curl-9201-1a6e9d" Oct 30 05:55:51.676673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.676597 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" containerName="test-curl-9201-1a6e9d" Oct 30 05:55:51.680760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.680739 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.696284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.696259 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz"] Oct 30 05:55:51.701721 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.701698 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:55:51.701872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.701731 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:55:51.701984 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.701876 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:55:51.711446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.711427 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:55:51.782944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.782901 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ksn6\" (UniqueName: \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.783535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.783024 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.783535 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.783155 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.885026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.884993 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ksn6\" (UniqueName: \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.885214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.885101 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.885214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.885162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.885785 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.885762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.887610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.887589 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:51.910307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:51.910277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ksn6\" (UniqueName: \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\") pod \"maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:52.000969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:52.000940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:52.085740 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:52.085702 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9201-1a6e9d" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" containerName="test-curl-9201-1a6e9d" containerID="cri-o://c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6" gracePeriod=2 Oct 30 05:55:52.472487 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:52.446913 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz"] Oct 30 05:55:53.099342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:53.099258 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" event={"ID":"2c0de957-a41f-4038-8f6a-8e86e4720080","Type":"ContainerStarted","Data":"f01a80008449db314afcf0b6cfc2d192c45500a953d3c6d38c514cf50d9f457a"} Oct 30 05:55:53.099342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:53.099323 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" event={"ID":"2c0de957-a41f-4038-8f6a-8e86e4720080","Type":"ContainerStarted","Data":"de50a0084647e9a7082174594fc1b8b0df544980fb706d4d467c12c0fe6277e0"} Oct 30 05:55:53.148823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:53.148692 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" podStartSLOduration=2.148672312 podStartE2EDuration="2.148672312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:53.144465127 +0000 UTC m=+1411.584842324" watchObservedRunningTime="2026-10-30 05:55:53.148672312 +0000 UTC m=+1411.589049507" Oct 30 05:55:54.113503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.113479 6171 generic.go:356] "Generic (PLEG): container finished" podID="2c0de957-a41f-4038-8f6a-8e86e4720080" containerID="f01a80008449db314afcf0b6cfc2d192c45500a953d3c6d38c514cf50d9f457a" exitCode=0 Oct 30 05:55:54.113862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.113564 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" event={"ID":"2c0de957-a41f-4038-8f6a-8e86e4720080","Type":"ContainerDied","Data":"f01a80008449db314afcf0b6cfc2d192c45500a953d3c6d38c514cf50d9f457a"} Oct 30 05:55:54.489607 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.489581 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:54.524764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.524738 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\" (UniqueName: \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\") pod \"0dc8e44e-7877-496b-9a4b-8a171c3de7a2\" (UID: \"0dc8e44e-7877-496b-9a4b-8a171c3de7a2\") " Oct 30 05:55:54.526970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.526944 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw" pod "0dc8e44e-7877-496b-9a4b-8a171c3de7a2" (UID: "0dc8e44e-7877-496b-9a4b-8a171c3de7a2"). InnerVolumeSpecName "kube-api-access-t7wtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:54.627320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:54.627236 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wtw\" (UniqueName: \"kubernetes.io/projected/0dc8e44e-7877-496b-9a4b-8a171c3de7a2-kube-api-access-t7wtw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.126969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.126934 6171 generic.go:356] "Generic (PLEG): container finished" podID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" containerID="c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6" exitCode=137 Oct 30 05:55:55.127439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.126976 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:55.127439 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.127034 6171 scope.go:122] "RemoveContainer" containerID="c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6" Oct 30 05:55:55.144578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.144561 6171 scope.go:122] "RemoveContainer" containerID="c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6" Oct 30 05:55:55.144874 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:55:55.144838 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6\": container with ID starting with c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6 not found: ID does not exist" containerID="c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6" Oct 30 05:55:55.144972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.144882 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6"} err="failed to get container status \"c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6\": rpc error: code = NotFound desc = could not find container \"c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6\": container with ID starting with c786cab9bd9cb42df44e1a04e07bccee86f55c6731406860fcf70ae96302f0e6 not found: ID does not exist" Oct 30 05:55:55.158058 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:55:55.157991 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9201-1a6e9d\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:55.321832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.321810 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:55.384950 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:55:55.384856 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9201-1a6e9d\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" pod="opendatahub/test-curl-9201-1a6e9d" Oct 30 05:55:55.440937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.440907 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\") pod \"2c0de957-a41f-4038-8f6a-8e86e4720080\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " Oct 30 05:55:55.441141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.441093 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\" (UniqueName: \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\") pod \"2c0de957-a41f-4038-8f6a-8e86e4720080\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " Oct 30 05:55:55.441141 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.441125 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\") pod \"2c0de957-a41f-4038-8f6a-8e86e4720080\" (UID: \"2c0de957-a41f-4038-8f6a-8e86e4720080\") " Oct 30 05:55:55.441616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.441587 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca" pod "2c0de957-a41f-4038-8f6a-8e86e4720080" (UID: "2c0de957-a41f-4038-8f6a-8e86e4720080"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:55.442368 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.442342 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.443308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.443278 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca" pod "2c0de957-a41f-4038-8f6a-8e86e4720080" (UID: "2c0de957-a41f-4038-8f6a-8e86e4720080"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:55:55.443479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.443457 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6" pod "2c0de957-a41f-4038-8f6a-8e86e4720080" (UID: "2c0de957-a41f-4038-8f6a-8e86e4720080"). InnerVolumeSpecName "kube-api-access-2ksn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:55.544039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.544013 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ksn6\" (UniqueName: \"kubernetes.io/projected/2c0de957-a41f-4038-8f6a-8e86e4720080-kube-api-access-2ksn6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.544186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.544079 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c0de957-a41f-4038-8f6a-8e86e4720080-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:55:55.676270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:55.676196 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dc8e44e-7877-496b-9a4b-8a171c3de7a2" path="/var/lib/kubelet/pods/0dc8e44e-7877-496b-9a4b-8a171c3de7a2/volumes" Oct 30 05:55:56.040214 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.040181 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk"] Oct 30 05:55:56.046789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.046753 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t"] Oct 30 05:55:56.053462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.053437 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b395cac9-d6qpk"] Oct 30 05:55:56.059001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.058977 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5f50b80f-fsf5t"] Oct 30 05:55:56.063956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.063937 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z"] Oct 30 05:55:56.068863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.068837 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0aae7c2-2jh7z"] Oct 30 05:55:56.139967 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.139948 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" Oct 30 05:55:56.140326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.139942 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz" event={"ID":"2c0de957-a41f-4038-8f6a-8e86e4720080","Type":"ContainerDied","Data":"de50a0084647e9a7082174594fc1b8b0df544980fb706d4d467c12c0fe6277e0"} Oct 30 05:55:56.140326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:56.140111 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de50a0084647e9a7082174594fc1b8b0df544980fb706d4d467c12c0fe6277e0" Oct 30 05:55:56.383986 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:55:56.383901 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:55:57.676119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:57.676076 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10ed4255-f11d-4ac4-ab20-d476f532db79" path="/var/lib/kubelet/pods/10ed4255-f11d-4ac4-ab20-d476f532db79/volumes" Oct 30 05:55:57.676893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:57.676868 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74a52252-82ac-4256-919d-9c065c3aab64" path="/var/lib/kubelet/pods/74a52252-82ac-4256-919d-9c065c3aab64/volumes" Oct 30 05:55:57.677482 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:55:57.677456 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59d0e90-3b18-403f-bb1d-ff82b228d3e5" path="/var/lib/kubelet/pods/a59d0e90-3b18-403f-bb1d-ff82b228d3e5/volumes" Oct 30 05:56:02.045231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:02.045190 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v"] Oct 30 05:56:02.058948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:02.058912 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-466e6410-lgf5v"] Oct 30 05:56:03.819804 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:03.819764 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06de76a3-f362-4a1a-aae2-663e55a185de" path="/var/lib/kubelet/pods/06de76a3-f362-4a1a-aae2-663e55a185de/volumes" Oct 30 05:56:05.033683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:05.033651 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc"] Oct 30 05:56:05.046847 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:05.046822 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58c83b18-dn8qc"] Oct 30 05:56:05.676825 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:05.676789 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="396ea7be-2d9c-480e-b3eb-fecb7e47e379" path="/var/lib/kubelet/pods/396ea7be-2d9c-480e-b3eb-fecb7e47e379/volumes" Oct 30 05:56:06.047087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:06.047026 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m"] Oct 30 05:56:06.060784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:06.060753 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-461da871-67j5m"] Oct 30 05:56:07.676511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:07.676469 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a8a75e6-7355-48fa-8a01-d0d0409e638c" path="/var/lib/kubelet/pods/9a8a75e6-7355-48fa-8a01-d0d0409e638c/volumes" Oct 30 05:56:19.043284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:19.043244 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84"] Oct 30 05:56:19.056965 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:19.056936 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8df2ee44-54j84"] Oct 30 05:56:19.676913 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:19.676868 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="952c2fa8-520f-4e58-9265-49463debc9e8" path="/var/lib/kubelet/pods/952c2fa8-520f-4e58-9265-49463debc9e8/volumes" Oct 30 05:56:20.041420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:20.041380 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5"] Oct 30 05:56:20.047491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:20.047462 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28004554-hf7p5"] Oct 30 05:56:20.065536 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:20.065507 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:56:22.579195 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.579141 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:22.735707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.735615 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efea9f9d-5603-4c46-a2a1-746102dcb82f" path="/var/lib/kubelet/pods/efea9f9d-5603-4c46-a2a1-746102dcb82f/volumes" Oct 30 05:56:22.736674 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:56:22.736526 6171 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.062s" Oct 30 05:56:22.742266 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.742237 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:56:22.753259 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.753219 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" containerID="cri-o://817fbf830bfd087ff4a222519ad23e313c402fc6f76be4627434d8accb61d571" gracePeriod=30 Oct 30 05:56:22.761499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.761470 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:56:22.761928 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.761901 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" containerID="cri-o://71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644" gracePeriod=30 Oct 30 05:56:22.773192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.773158 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2"] Oct 30 05:56:22.794408 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.794372 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c8a7907-j5pm2"] Oct 30 05:56:22.847978 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.847895 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:56:22.848153 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.848096 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" containerID="cri-o://538cd677d55f215392c79262a74a13372260e97932c40e56c623f3f8fc563c52" gracePeriod=30 Oct 30 05:56:22.855922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.855899 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:56:22.856088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.856068 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" containerID="cri-o://6bb8c43e6d7f849589c814a5dae99846f40e8c8316ceca295d16248573790acf" gracePeriod=30 Oct 30 05:56:22.863032 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.863007 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:56:22.863273 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:22.863250 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" containerID="cri-o://184f75a6035be7db3535e1a9508141e7fb640cea8cecbb66a5f43cab8c20cad6" gracePeriod=30 Oct 30 05:56:23.676111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:23.676078 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f70eb699-96ee-43e2-bc52-d3eb63dfe83a" path="/var/lib/kubelet/pods/f70eb699-96ee-43e2-bc52-d3eb63dfe83a/volumes" Oct 30 05:56:24.164080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.164001 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd"] Oct 30 05:56:24.170554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.170519 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3917d7c-25tgd"] Oct 30 05:56:24.365269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.365239 6171 scope.go:122] "RemoveContainer" containerID="f2c4a87f236c59680bffda9223517157b0a8c58d324ad09b55371e324e95cf6a" Oct 30 05:56:24.377372 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.377352 6171 scope.go:122] "RemoveContainer" containerID="946086a43f45db72cb1159047a1f647d6ff390a14a05af583011c887ec675278" Oct 30 05:56:24.434936 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.434913 6171 scope.go:122] "RemoveContainer" containerID="bc8ce0a370546ead4dbd37b8abc0c0cf27e52fc4b5eb71ad64bf332b1482cab1" Oct 30 05:56:24.446569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.446541 6171 scope.go:122] "RemoveContainer" containerID="30ab555e6a959d064fab66f844699afbf9804d0a1f8366980bcf5cc7278d7d9b" Oct 30 05:56:24.457646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.457624 6171 scope.go:122] "RemoveContainer" containerID="9ee52fcbbbca55553b82dda3c6e506799ee43f8c7a799939c690273032dda8b9" Oct 30 05:56:24.473788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.473774 6171 scope.go:122] "RemoveContainer" containerID="3324a174d57fbd844a803a998974f393a47039b8c585926489cc149be3a47c9e" Oct 30 05:56:24.540674 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.540645 6171 scope.go:122] "RemoveContainer" containerID="8ca0dc36f616dbc5510500d700ea923575a3a7faf2b3d9f1cfb1d524c4120bd9" Oct 30 05:56:24.552739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.552722 6171 scope.go:122] "RemoveContainer" containerID="d8e04a79e4dd899655a2c5443074dab745fd5227a776c952596481bc130e7f89" Oct 30 05:56:24.565022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.565005 6171 scope.go:122] "RemoveContainer" containerID="cbfc96a29f9c8fc9d4837747ce320cfb6b5ae88754c8788d1449e5f44823ecb2" Oct 30 05:56:24.599570 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.599191 6171 scope.go:122] "RemoveContainer" containerID="75cb1bb940741d3185018cda7addee9b39bc5913c0d15d57bb19582ddf390d76" Oct 30 05:56:24.647808 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.647782 6171 scope.go:122] "RemoveContainer" containerID="50533096d698dcf73ede09f2bfbff290c9f4515311d1f8f60185db56df177251" Oct 30 05:56:24.661907 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:24.661890 6171 scope.go:122] "RemoveContainer" containerID="a678e875e8de3b941566e900205df5f6b99cefa06bd98968b1dc1f2af12dead4" Oct 30 05:56:25.676932 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:25.676898 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="811f8d90-1c5f-4621-a21e-aa77b673230d" path="/var/lib/kubelet/pods/811f8d90-1c5f-4621-a21e-aa77b673230d/volumes" Oct 30 05:56:26.431281 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:56:26.431246 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:56:28.313126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.311031 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body= Oct 30 05:56:28.313126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.311149 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" Oct 30 05:56:28.451695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.450482 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:56:28.672132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.667960 6171 generic.go:356] "Generic (PLEG): container finished" podID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerID="184f75a6035be7db3535e1a9508141e7fb640cea8cecbb66a5f43cab8c20cad6" exitCode=0 Oct 30 05:56:28.672132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.668195 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" event={"ID":"651a1fce-06de-4467-8bcd-170ebc160cd5","Type":"ContainerDied","Data":"184f75a6035be7db3535e1a9508141e7fb640cea8cecbb66a5f43cab8c20cad6"} Oct 30 05:56:28.672132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.670800 6171 generic.go:356] "Generic (PLEG): container finished" podID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerID="6bb8c43e6d7f849589c814a5dae99846f40e8c8316ceca295d16248573790acf" exitCode=0 Oct 30 05:56:28.672132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.670991 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" event={"ID":"34715cfc-378c-4447-88a8-4c271e37c2f1","Type":"ContainerDied","Data":"6bb8c43e6d7f849589c814a5dae99846f40e8c8316ceca295d16248573790acf"} Oct 30 05:56:28.704073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.676644 6171 generic.go:356] "Generic (PLEG): container finished" podID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerID="71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644" exitCode=0 Oct 30 05:56:28.704073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.676782 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" event={"ID":"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8","Type":"ContainerDied","Data":"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644"} Oct 30 05:56:28.704073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.676823 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" event={"ID":"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8","Type":"ContainerDied","Data":"fa0fcdf4152b7a5f7275527c1c7c22e2cc99496314a62311fedd7abebb8fa187"} Oct 30 05:56:28.704073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.676852 6171 scope.go:122] "RemoveContainer" containerID="71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644" Oct 30 05:56:28.704073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.677019 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7" Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.765734 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.765938 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.766023 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.766192 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.766395 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.766486 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.779992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.777255 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:28.794269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.794122 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:28.794577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.794550 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.819479 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.820936 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2c0de957-a41f-4038-8f6a-8e86e4720080" containerName="revoke-keys" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.820954 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0de957-a41f-4038-8f6a-8e86e4720080" containerName="revoke-keys" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.820993 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.821001 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.821271 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0de957-a41f-4038-8f6a-8e86e4720080" containerName="revoke-keys" Oct 30 05:56:28.823954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.821293 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" containerName="istio-proxy" Oct 30 05:56:28.826429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.825570 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:28.826429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.825644 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:28.826429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.825650 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "kube-api-access-d9k7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:28.830583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.830437 6171 generic.go:356] "Generic (PLEG): container finished" podID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerID="817fbf830bfd087ff4a222519ad23e313c402fc6f76be4627434d8accb61d571" exitCode=0 Oct 30 05:56:28.832898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.832835 6171 generic.go:356] "Generic (PLEG): container finished" podID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerID="538cd677d55f215392c79262a74a13372260e97932c40e56c623f3f8fc563c52" exitCode=0 Oct 30 05:56:28.855871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.855830 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 05:56:28.857219 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.857196 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" event={"ID":"63075e82-f898-431d-aca8-a6cea2a94cb7","Type":"ContainerDied","Data":"817fbf830bfd087ff4a222519ad23e313c402fc6f76be4627434d8accb61d571"} Oct 30 05:56:28.857517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.857383 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" event={"ID":"41b9538b-7e10-49ba-9cd4-d2e188f2742a","Type":"ContainerDied","Data":"538cd677d55f215392c79262a74a13372260e97932c40e56c623f3f8fc563c52"} Oct 30 05:56:28.857649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.857138 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:28.875267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.875192 6171 scope.go:122] "RemoveContainer" containerID="71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644" Oct 30 05:56:28.876787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.876480 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:56:28.876787 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.876687 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4vtg8\"" Oct 30 05:56:28.877701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.877024 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-862dc2-serving-cert\"" Oct 30 05:56:28.877701 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:56:28.877618 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644\": container with ID starting with 71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644 not found: ID does not exist" containerID="71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644" Oct 30 05:56:28.877701 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.877646 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644"} err="failed to get container status \"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644\": rpc error: code = NotFound desc = could not find container \"71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644\": container with ID starting with 71cd57b26444cdd8e9a3d646455383b8c275e922dbd5ae8e0a13f4b01f213644 not found: ID does not exist" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.879803 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.879847 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.880021 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\") pod \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\" (UID: \"ddfa14d2-2219-4b21-9756-a6ff0b3d38d8\") " Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881142 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881161 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881176 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k7f\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-kube-api-access-d9k7f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881190 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881204 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.881397 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881217 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.882171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.881962 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:28.882654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.882375 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:28.889699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.889678 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:56:28.889778 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.889768 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:56:28.892849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.891890 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token" pod "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" (UID: "ddfa14d2-2219-4b21-9756-a6ff0b3d38d8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:28.976801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.976776 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:56:28.980597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.980576 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:56:28.982556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.982534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:28.982673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.982656 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:28.982742 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.982698 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88vfn\" (UniqueName: \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:28.982801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.982742 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:28.983093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.983031 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.983203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.983095 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.983203 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.983108 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:28.985507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.985488 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:56:28.990223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:28.990203 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:56:29.046896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.046866 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:56:29.048298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048277 6171 cpu_manager.go:414] "containerMap: removing container" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" Oct 30 05:56:29.048394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048300 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" Oct 30 05:56:29.048394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048327 6171 cpu_manager.go:414] "containerMap: removing container" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" Oct 30 05:56:29.048394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048335 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" Oct 30 05:56:29.048525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048419 6171 cpu_manager.go:414] "containerMap: removing container" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" Oct 30 05:56:29.048525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048427 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" Oct 30 05:56:29.048525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048460 6171 cpu_manager.go:414] "containerMap: removing container" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" Oct 30 05:56:29.048525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048467 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" Oct 30 05:56:29.048719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048663 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" containerName="istio-proxy" Oct 30 05:56:29.048719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048680 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" containerName="istio-proxy" Oct 30 05:56:29.048719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048697 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" Oct 30 05:56:29.048719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.048709 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" Oct 30 05:56:29.057264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.054846 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.078105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.078078 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a3efab09-serving-cert\"" Oct 30 05:56:29.078488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.078444 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:56:29.085122 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.085099 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.085593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.085490 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.089158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.089136 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.089592 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.089489 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.089751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.089734 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.090279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.090183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:29.090813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.090796 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.091011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.090996 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.091546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.091530 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.092207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.092188 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.092788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.092762 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.093260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.092659 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.093588 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.093561 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.093705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.093205 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.093836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.093823 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.094347 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.094322 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.094835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.094265 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:29.094835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.094714 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.095295 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095038 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.095430 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095415 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.095547 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095532 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.095671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095658 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.095826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095812 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.095982 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095967 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096249 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095364 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096316 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096535 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096573 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096608 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096688 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096791 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096855 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096914 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096957 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097021 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097079 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097110 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097194 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\") pod \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\" (UID: \"41b9538b-7e10-49ba-9cd4-d2e188f2742a\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097229 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\") pod \"651a1fce-06de-4467-8bcd-170ebc160cd5\" (UID: \"651a1fce-06de-4467-8bcd-170ebc160cd5\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097258 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097309 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\") pod \"34715cfc-378c-4447-88a8-4c271e37c2f1\" (UID: \"34715cfc-378c-4447-88a8-4c271e37c2f1\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097354 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\") pod \"63075e82-f898-431d-aca8-a6cea2a94cb7\" (UID: \"63075e82-f898-431d-aca8-a6cea2a94cb7\") " Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098254 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098429 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098448 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88vfn\" (UniqueName: \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098478 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098606 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098616 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098626 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098634 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098644 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098651 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63075e82-f898-431d-aca8-a6cea2a94cb7-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098659 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.099070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098667 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095750 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.095869 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "kube-api-access-78b8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096442 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.096642 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097784 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.097968 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098175 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.098369 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.099236 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.101350 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.101799 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.101864 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.101871 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.103264 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "kube-api-access-c6fdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.103312 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:29.104357 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.103964 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:29.105304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.104675 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.106410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.105812 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.107903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.107876 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.112918 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.112805 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.113720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.113650 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.116083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.115539 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "kube-api-access-x9bz5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.116083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.115621 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.116083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.115673 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "kube-api-access-bhmm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.116083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.115718 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy" pod "63075e82-f898-431d-aca8-a6cea2a94cb7" (UID: "63075e82-f898-431d-aca8-a6cea2a94cb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.116083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.115764 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:29.116410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.116192 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.118391 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.118364 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:56:29.122509 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.122484 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.129188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.126577 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.136093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.136003 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88vfn\" (UniqueName: \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\") pod \"maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.146478 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.146455 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:56:29.150582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.150542 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy" pod "41b9538b-7e10-49ba-9cd4-d2e188f2742a" (UID: "41b9538b-7e10-49ba-9cd4-d2e188f2742a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:29.154077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.154023 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo" pod "651a1fce-06de-4467-8bcd-170ebc160cd5" (UID: "651a1fce-06de-4467-8bcd-170ebc160cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:29.154321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.154290 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token" pod "34715cfc-378c-4447-88a8-4c271e37c2f1" (UID: "34715cfc-378c-4447-88a8-4c271e37c2f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:29.170156 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.168887 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 05:56:29.178355 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.178329 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-b202a0-serving-cert\"" Oct 30 05:56:29.186073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.184208 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 05:56:29.186073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.184292 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.200522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.200488 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkdn9\" (UniqueName: \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.201944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.200697 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.202087 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.201990 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.202161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202105 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.202297 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202277 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/34715cfc-378c-4447-88a8-4c271e37c2f1-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202305 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhmm2\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-kube-api-access-bhmm2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202319 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202332 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202345 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9bz5\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-kube-api-access-x9bz5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202358 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202380 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202372 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202384 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202397 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202410 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202423 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202436 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202449 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202462 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/34715cfc-378c-4447-88a8-4c271e37c2f1-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202475 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fdv\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-kube-api-access-c6fdv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202487 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202500 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202512 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202524 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202536 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202548 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/651a1fce-06de-4467-8bcd-170ebc160cd5-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202560 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202572 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/34715cfc-378c-4447-88a8-4c271e37c2f1-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202585 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202597 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202609 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/651a1fce-06de-4467-8bcd-170ebc160cd5-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202621 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78b8f\" (UniqueName: \"kubernetes.io/projected/63075e82-f898-431d-aca8-a6cea2a94cb7-kube-api-access-78b8f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.202738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.202634 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41b9538b-7e10-49ba-9cd4-d2e188f2742a-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:29.213307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.213285 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:56:29.218345 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.218329 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.257133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.250605 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:56:29.273748 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.273716 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:29.283298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.283268 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4fc75f-serving-cert\"" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316506 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316563 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316656 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316688 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316729 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316762 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316806 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316848 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.316957 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317093 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317159 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317223 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gftpj\" (UniqueName: \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317255 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317281 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317294 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317314 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv22j\" (UniqueName: \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317337 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317394 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkdn9\" (UniqueName: \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317428 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rnst\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.317446 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.322181 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.318261 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.328842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.328760 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.336887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.336836 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.350776 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.350750 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkdn9\" (UniqueName: \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\") pod \"maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.352681 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.352661 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:56:29.362333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.362308 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-mxxj7"] Oct 30 05:56:29.401159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.401129 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419481 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419567 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gftpj\" (UniqueName: \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419614 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419652 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419679 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419711 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv22j\" (UniqueName: \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419749 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419842 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rnst\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419867 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.419983 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420014 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420203 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420249 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420299 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420398 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.420679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420512 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.420850 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.421246 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.424437 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.424681 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.424827 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.425303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.424834 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.426610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.425871 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.426610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.426294 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.427008 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.426969 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.427452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.427380 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.428415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.428378 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.428955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.428915 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.429505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.429483 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.451572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.451120 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.452724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.452659 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv22j\" (UniqueName: \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\") pod \"maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.464099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.462657 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gftpj\" (UniqueName: \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\") pod \"maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.467084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.467057 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:29.468490 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.467408 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rnst\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\") pod \"e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.522661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.519824 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:29.558165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.550918 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:29.644689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.641929 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 05:56:29.650998 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:56:29.650964 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22dec94d_c650_478e_b581_63ae985bca08.slice/crio-26d6723865b4c930d6265e28e8411af6c368642a837f9d2a7f75bed85fcce559 WatchSource:0}: Error finding container 26d6723865b4c930d6265e28e8411af6c368642a837f9d2a7f75bed85fcce559: Status 404 returned error can't find the container with id 26d6723865b4c930d6265e28e8411af6c368642a837f9d2a7f75bed85fcce559 Oct 30 05:56:29.654309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.652793 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:56:29.654309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.652850 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:29.689015 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.688954 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfa14d2-2219-4b21-9756-a6ff0b3d38d8" path="/var/lib/kubelet/pods/ddfa14d2-2219-4b21-9756-a6ff0b3d38d8/volumes" Oct 30 05:56:29.860067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.858131 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:56:29.860067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.858205 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:29.912486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.912453 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" Oct 30 05:56:29.912987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.912452 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf" event={"ID":"41b9538b-7e10-49ba-9cd4-d2e188f2742a","Type":"ContainerDied","Data":"f0cbb13f2e6bf1784bf48ba4d2d64247f1042f43813a666f0a26e3320779c680"} Oct 30 05:56:29.912987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.912880 6171 scope.go:122] "RemoveContainer" containerID="538cd677d55f215392c79262a74a13372260e97932c40e56c623f3f8fc563c52" Oct 30 05:56:29.915395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.914996 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" event={"ID":"651a1fce-06de-4467-8bcd-170ebc160cd5","Type":"ContainerDied","Data":"be252b5e8f983ba4c4d9d1e45f21c28fae2fea5fe05c6e280c6c2ef0773bdc9b"} Oct 30 05:56:29.915395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.915140 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz" Oct 30 05:56:29.917790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.917396 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" event={"ID":"22dec94d-c650-478e-b581-63ae985bca08","Type":"ContainerStarted","Data":"26d6723865b4c930d6265e28e8411af6c368642a837f9d2a7f75bed85fcce559"} Oct 30 05:56:29.919493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.919466 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" event={"ID":"34715cfc-378c-4447-88a8-4c271e37c2f1","Type":"ContainerDied","Data":"e956b9898a35d90067c33537de651f85d44c13722af912f6a5ebad2b2aeafff6"} Oct 30 05:56:29.919671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.919658 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf" Oct 30 05:56:29.929409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.928028 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" event={"ID":"63075e82-f898-431d-aca8-a6cea2a94cb7","Type":"ContainerDied","Data":"b870b3da10993e51449a59af9307570d89878acdee9a367aa9dfa9cb1e8657a4"} Oct 30 05:56:29.929409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.928210 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd" Oct 30 05:56:29.932729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.932714 6171 scope.go:122] "RemoveContainer" containerID="184f75a6035be7db3535e1a9508141e7fb640cea8cecbb66a5f43cab8c20cad6" Oct 30 05:56:29.956993 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:56:29.956968 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80559662_d089_4324_898e_7563c22c4d4b.slice/crio-36aa26b88dfe35480599a96a76996ab40ad00e8e9ea6aafb07b7b8734558fb26 WatchSource:0}: Error finding container 36aa26b88dfe35480599a96a76996ab40ad00e8e9ea6aafb07b7b8734558fb26: Status 404 returned error can't find the container with id 36aa26b88dfe35480599a96a76996ab40ad00e8e9ea6aafb07b7b8734558fb26 Oct 30 05:56:29.958663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.958640 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:56:29.964479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.964447 6171 scope.go:122] "RemoveContainer" containerID="6bb8c43e6d7f849589c814a5dae99846f40e8c8316ceca295d16248573790acf" Oct 30 05:56:29.996145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.993375 6171 scope.go:122] "RemoveContainer" containerID="817fbf830bfd087ff4a222519ad23e313c402fc6f76be4627434d8accb61d571" Oct 30 05:56:29.998457 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:29.998376 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:56:30.007688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.007665 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-kkdwf"] Oct 30 05:56:30.040360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.040326 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:56:30.075068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.074986 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:56:30.147340 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:56:30.147305 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod371c1e7c_c543_44df_b781_08b79e15ecb7.slice/crio-384165c8ece3af4e13f66ac1f689f0a954c59268061c7d8fac31f527ae7d3c54 WatchSource:0}: Error finding container 384165c8ece3af4e13f66ac1f689f0a954c59268061c7d8fac31f527ae7d3c54: Status 404 returned error can't find the container with id 384165c8ece3af4e13f66ac1f689f0a954c59268061c7d8fac31f527ae7d3c54 Oct 30 05:56:30.218820 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.218791 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-mtvqz"] Oct 30 05:56:30.226524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.226030 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:30.226524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.226144 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:30.226524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.226192 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:30.237869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.237813 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:56:30.247877 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.247649 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-5j4vf"] Oct 30 05:56:30.258340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.258189 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 05:56:30.265543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.265484 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:56:30.273828 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.273806 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-5blxd"] Oct 30 05:56:30.281949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.281930 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:56:30.985611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.985573 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" event={"ID":"80559662-d089-4324-898e-7563c22c4d4b","Type":"ContainerStarted","Data":"36aa26b88dfe35480599a96a76996ab40ad00e8e9ea6aafb07b7b8734558fb26"} Oct 30 05:56:30.993101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.992282 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" event={"ID":"de0584cf-4f5a-4280-ae92-22c2cdf5e060","Type":"ContainerStarted","Data":"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239"} Oct 30 05:56:30.993101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:30.992332 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" event={"ID":"de0584cf-4f5a-4280-ae92-22c2cdf5e060","Type":"ContainerStarted","Data":"5683473c37ee5968c969705d9c9a4ea75cd5ca6315cd22a27cda4ab0a3c3550a"} Oct 30 05:56:31.007711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.007648 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" event={"ID":"371c1e7c-c543-44df-b781-08b79e15ecb7","Type":"ContainerStarted","Data":"384165c8ece3af4e13f66ac1f689f0a954c59268061c7d8fac31f527ae7d3c54"} Oct 30 05:56:31.016013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.015940 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" event={"ID":"3d35d538-9e00-40d2-aecb-8ed882773fed","Type":"ContainerStarted","Data":"e171275ead755414914d244c96de276e7d7022e137922cf90e08f0b538ee550a"} Oct 30 05:56:31.059354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.059301 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podStartSLOduration=47.059283456 podStartE2EDuration="47.059283456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:31.043739922 +0000 UTC m=+1449.484117116" watchObservedRunningTime="2026-10-30 05:56:31.059283456 +0000 UTC m=+1449.499660650" Oct 30 05:56:31.556517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.556477 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:31.570067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.566223 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:31.570067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.566298 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:31.741104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.741068 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34715cfc-378c-4447-88a8-4c271e37c2f1" path="/var/lib/kubelet/pods/34715cfc-378c-4447-88a8-4c271e37c2f1/volumes" Oct 30 05:56:31.742169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.742133 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41b9538b-7e10-49ba-9cd4-d2e188f2742a" path="/var/lib/kubelet/pods/41b9538b-7e10-49ba-9cd4-d2e188f2742a/volumes" Oct 30 05:56:31.743027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.743008 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63075e82-f898-431d-aca8-a6cea2a94cb7" path="/var/lib/kubelet/pods/63075e82-f898-431d-aca8-a6cea2a94cb7/volumes" Oct 30 05:56:31.744286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:31.744268 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651a1fce-06de-4467-8bcd-170ebc160cd5" path="/var/lib/kubelet/pods/651a1fce-06de-4467-8bcd-170ebc160cd5/volumes" Oct 30 05:56:32.557766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.557320 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:32.557766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.557382 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:32.854602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.853600 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6"] Oct 30 05:56:32.864623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.863716 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:32.864623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.864599 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6"] Oct 30 05:56:32.895759 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.895732 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:56:32.896253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.896214 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:56:32.959975 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.959938 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:32.960332 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.960302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4jl\" (UniqueName: \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:32.960558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:32.960534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.040474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.040409 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6"] Oct 30 05:56:33.052774 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.052708 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f"] Oct 30 05:56:33.063606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.062516 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.063606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.062629 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4jl\" (UniqueName: \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.063606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.062832 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.066574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.066415 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.078034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.077980 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-40a76272-5k7p6"] Oct 30 05:56:33.105174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.082464 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.120484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.120452 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-343574e2-zk87f"] Oct 30 05:56:33.126459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.126401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4jl\" (UniqueName: \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\") pod \"maas-api-revoke-sub-1135c258-llwn6\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.211073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.205351 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:33.557298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.556467 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:33.557298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.556525 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:33.728223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.728183 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a9ce81-4778-4883-b74d-e81397c12c58" path="/var/lib/kubelet/pods/32a9ce81-4778-4883-b74d-e81397c12c58/volumes" Oct 30 05:56:33.729009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:33.728983 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804fca7b-2f85-4ac9-a9df-a0e81f9afedd" path="/var/lib/kubelet/pods/804fca7b-2f85-4ac9-a9df-a0e81f9afedd/volumes" Oct 30 05:56:34.019794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:34.019759 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6"] Oct 30 05:56:34.141340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:34.141260 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" event={"ID":"4a1ae32d-416f-43fe-b716-f6508e2ac261","Type":"ContainerStarted","Data":"45c17af12b473472ce8abd06537ae04d0cd7e0807ea22d9646639f4ca2ed271a"} Oct 30 05:56:34.558079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:34.556499 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:34.558079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:34.556556 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:35.132806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.132772 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4"] Oct 30 05:56:35.145125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.145093 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c3de0bdc-hwfw4"] Oct 30 05:56:35.163753 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.163709 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" event={"ID":"4a1ae32d-416f-43fe-b716-f6508e2ac261","Type":"ContainerStarted","Data":"9fcd70c13398bc400a7fa40576333d0979f50609f721573cfcd8b231c34c40e1"} Oct 30 05:56:35.228509 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.228444 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" podStartSLOduration=3.228424498 podStartE2EDuration="3.228424498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:35.22143156 +0000 UTC m=+1453.661808753" watchObservedRunningTime="2026-10-30 05:56:35.228424498 +0000 UTC m=+1453.668801693" Oct 30 05:56:35.560725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.559434 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:35.560725 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.559499 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:35.710243 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:35.710204 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc81b47-a7d9-4b94-9241-a2f664ffd523" path="/var/lib/kubelet/pods/4cc81b47-a7d9-4b94-9241-a2f664ffd523/volumes" Oct 30 05:56:36.041629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.041587 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn"] Oct 30 05:56:36.052611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.052579 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0690fdaf-kg8xn"] Oct 30 05:56:36.213905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.213875 6171 generic.go:356] "Generic (PLEG): container finished" podID="4a1ae32d-416f-43fe-b716-f6508e2ac261" containerID="9fcd70c13398bc400a7fa40576333d0979f50609f721573cfcd8b231c34c40e1" exitCode=0 Oct 30 05:56:36.214334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.214011 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" event={"ID":"4a1ae32d-416f-43fe-b716-f6508e2ac261","Type":"ContainerDied","Data":"9fcd70c13398bc400a7fa40576333d0979f50609f721573cfcd8b231c34c40e1"} Oct 30 05:56:36.557849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.557669 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" start-of-body= Oct 30 05:56:36.557849 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:36.557729 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.194:15021/healthz/ready\": dial tcp 10.133.0.194:15021: connect: connection refused" Oct 30 05:56:37.595094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.595063 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:37.596939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.596912 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:37.598159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.598138 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:56:37.680486 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.680451 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f" path="/var/lib/kubelet/pods/0bcf2f3a-6559-42ea-a44e-f0e5abe6bd4f/volumes" Oct 30 05:56:37.726184 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.725904 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:37.756511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.756435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\") pod \"4a1ae32d-416f-43fe-b716-f6508e2ac261\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " Oct 30 05:56:37.756707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.756530 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\") pod \"4a1ae32d-416f-43fe-b716-f6508e2ac261\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " Oct 30 05:56:37.756707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.756664 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\" (UniqueName: \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\") pod \"4a1ae32d-416f-43fe-b716-f6508e2ac261\" (UID: \"4a1ae32d-416f-43fe-b716-f6508e2ac261\") " Oct 30 05:56:37.758682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.758623 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca" pod "4a1ae32d-416f-43fe-b716-f6508e2ac261" (UID: "4a1ae32d-416f-43fe-b716-f6508e2ac261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:37.769461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.769383 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl" pod "4a1ae32d-416f-43fe-b716-f6508e2ac261" (UID: "4a1ae32d-416f-43fe-b716-f6508e2ac261"). InnerVolumeSpecName "kube-api-access-lt4jl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:37.770155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.770123 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca" pod "4a1ae32d-416f-43fe-b716-f6508e2ac261" (UID: "4a1ae32d-416f-43fe-b716-f6508e2ac261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:56:37.866387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.866353 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4jl\" (UniqueName: \"kubernetes.io/projected/4a1ae32d-416f-43fe-b716-f6508e2ac261-kube-api-access-lt4jl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:37.866387 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.866389 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:38.190538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:37.866407 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4a1ae32d-416f-43fe-b716-f6508e2ac261-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:56:38.341169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.341126 6171 generic.go:356] "Generic (PLEG): container finished" podID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerID="9305a9e20ea8bc0efed20121f782dcadf74b0485f4b769569c594315fb3717e4" exitCode=0 Oct 30 05:56:38.341364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.341277 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerDied","Data":"9305a9e20ea8bc0efed20121f782dcadf74b0485f4b769569c594315fb3717e4"} Oct 30 05:56:38.341973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.341941 6171 scope.go:122] "RemoveContainer" containerID="9305a9e20ea8bc0efed20121f782dcadf74b0485f4b769569c594315fb3717e4" Oct 30 05:56:38.349287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.349256 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" Oct 30 05:56:38.349287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.349265 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6" event={"ID":"4a1ae32d-416f-43fe-b716-f6508e2ac261","Type":"ContainerDied","Data":"45c17af12b473472ce8abd06537ae04d0cd7e0807ea22d9646639f4ca2ed271a"} Oct 30 05:56:38.349453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.349324 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45c17af12b473472ce8abd06537ae04d0cd7e0807ea22d9646639f4ca2ed271a" Oct 30 05:56:38.352339 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.352291 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:38.352479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.352457 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:38.352603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.352584 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:38.366470 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.366436 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": read tcp 10.133.0.2:58600->10.133.0.179:15021: read: connection reset by peer" start-of-body= Oct 30 05:56:38.366684 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:38.366641 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": read tcp 10.133.0.2:58600->10.133.0.179:15021: read: connection reset by peer" Oct 30 05:56:39.369252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.369215 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" event={"ID":"22dec94d-c650-478e-b581-63ae985bca08","Type":"ContainerStarted","Data":"8024a4e23239859786c996d82356b21ba39098ac3fb3891d831fcff44513e185"} Oct 30 05:56:39.369556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.369320 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:39.372699 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.372666 6171 generic.go:356] "Generic (PLEG): container finished" podID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerID="f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a" exitCode=0 Oct 30 05:56:39.372838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.372784 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerDied","Data":"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a"} Oct 30 05:56:39.373358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.373330 6171 scope.go:122] "RemoveContainer" containerID="f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a" Oct 30 05:56:39.377921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.377897 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerStarted","Data":"3827af6668b6aa61d86628baa9eea367d3c2fbc628922f0b7f9d6220515c086f"} Oct 30 05:56:39.407068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.405848 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:39.407068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.405956 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:39.407068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.406003 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:56:39.422408 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:39.422356 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" podStartSLOduration=28.924423021 podStartE2EDuration="38.422338803s" totalImagesPullingTime="9.497915782s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:39.417575426 +0000 UTC m=+1457.857952620" watchObservedRunningTime="2026-10-30 05:56:39.422338803 +0000 UTC m=+1457.862715997" Oct 30 05:56:40.323265 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.323159 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:56:40.324771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.324737 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:56:40.324903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.324792 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:56:40.423954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.423918 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" event={"ID":"371c1e7c-c543-44df-b781-08b79e15ecb7","Type":"ContainerStarted","Data":"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536"} Oct 30 05:56:40.424603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.424574 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:40.426216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.426188 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" event={"ID":"3d35d538-9e00-40d2-aecb-8ed882773fed","Type":"ContainerStarted","Data":"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f"} Oct 30 05:56:40.426369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.426351 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:40.428644 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.428622 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerStarted","Data":"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46"} Oct 30 05:56:40.430638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.430618 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" event={"ID":"80559662-d089-4324-898e-7563c22c4d4b","Type":"ContainerStarted","Data":"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1"} Oct 30 05:56:40.430747 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.430664 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:40.476736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.476679 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" podStartSLOduration=64.565047176 podStartE2EDuration="1m14.476667031s" totalImagesPullingTime="9.911619855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:40.472715331 +0000 UTC m=+1458.913092525" watchObservedRunningTime="2026-10-30 05:56:40.476667031 +0000 UTC m=+1458.917044257" Oct 30 05:56:40.623119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.622239 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" podStartSLOduration=43.232611847 podStartE2EDuration="52.62221709s" totalImagesPullingTime="9.389605243s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:40.607532531 +0000 UTC m=+1459.047909725" watchObservedRunningTime="2026-10-30 05:56:40.62221709 +0000 UTC m=+1459.062594283" Oct 30 05:56:40.702095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.689129 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:56:40.962147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:40.934347 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" podStartSLOduration=39.570554267 podStartE2EDuration="48.934280169s" totalImagesPullingTime="9.363725902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:40.703722079 +0000 UTC m=+1459.144099274" watchObservedRunningTime="2026-10-30 05:56:40.934280169 +0000 UTC m=+1459.374657363" Oct 30 05:56:41.331169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:41.330641 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:56:41.331169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:41.330703 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:56:41.729749 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:41.729712 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:41.730396 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:41.730366 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:41.978127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:41.978098 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:56:42.337687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:42.337647 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:56:42.337868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:42.337713 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:56:42.702123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:42.697147 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:42.702123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:42.697209 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:43.335071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:43.334532 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:56:43.335071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:43.334592 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:56:43.703205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:43.701419 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:43.703205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:43.701484 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:44.333887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:44.325154 6171 patch_prober.go:29] interesting pod/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" start-of-body= Oct 30 05:56:44.333887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:44.325216 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.187:15021/healthz/ready\": dial tcp 10.133.0.187:15021: connect: connection refused" Oct 30 05:56:44.693121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:44.689699 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:44.693121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:44.689852 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:45.343937 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:45.343541 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:56:45.345970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:45.345942 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 05:56:45.436013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:45.435983 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 05:56:45.707128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:45.691672 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:45.707128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:45.691728 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:46.038240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.038208 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562"] Oct 30 05:56:46.043480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.043453 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e66adadd-ql562"] Oct 30 05:56:46.515964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.515937 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 05:56:46.516388 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.516359 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:56:46.531241 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.531216 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:56:46.688185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.688150 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 05:56:46.688346 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:46.688241 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 05:56:47.675679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:47.675646 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ec6a329-44fe-40da-b9c9-21854045a52c" path="/var/lib/kubelet/pods/1ec6a329-44fe-40da-b9c9-21854045a52c/volumes" Oct 30 05:56:47.694371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:47.694343 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:56:47.694863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:47.694840 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:56:47.695544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:47.695517 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 05:56:49.040871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:49.039860 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng"] Oct 30 05:56:49.054633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:49.054607 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-325a4b2e-q7jng"] Oct 30 05:56:49.679290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:49.679258 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0" path="/var/lib/kubelet/pods/c50a9c97-4d9a-4f6f-8387-67ccb8a8a5b0/volumes" Oct 30 05:56:56.382583 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:56:56.382548 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:56:58.048557 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:58.048518 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn"] Oct 30 05:56:58.060144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:58.060116 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4325a71-9tzsn"] Oct 30 05:56:59.683549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:56:59.683515 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5a57c41-036a-4f5c-a7db-6f0ded6af417" path="/var/lib/kubelet/pods/f5a57c41-036a-4f5c-a7db-6f0ded6af417/volumes" Oct 30 05:57:03.894146 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.894109 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:03.895139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.895122 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4a1ae32d-416f-43fe-b716-f6508e2ac261" containerName="revoke-keys" Oct 30 05:57:03.895208 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.895141 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a1ae32d-416f-43fe-b716-f6508e2ac261" containerName="revoke-keys" Oct 30 05:57:03.895326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.895316 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a1ae32d-416f-43fe-b716-f6508e2ac261" containerName="revoke-keys" Oct 30 05:57:03.901261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.901240 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:03.924271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.924191 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:57:03.924271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.924231 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:57:03.931024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:03.930995 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:04.015895 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.015853 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vxm2\" (UniqueName: \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.016114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.016098 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.016255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.016234 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.118118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.118088 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.118267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.118203 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.118338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.118326 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vxm2\" (UniqueName: \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.118909 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.118889 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.120627 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.120610 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.143651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.143626 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vxm2\" (UniqueName: \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.231552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.231524 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:04.543080 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:04.543028 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8347028_f288_4b54_9174_d30374825a2f.slice/crio-2181a54b9452d594ecfb02fece8dacae01a4175679fd633bb6d0b7fc7bd98c3b WatchSource:0}: Error finding container 2181a54b9452d594ecfb02fece8dacae01a4175679fd633bb6d0b7fc7bd98c3b: Status 404 returned error can't find the container with id 2181a54b9452d594ecfb02fece8dacae01a4175679fd633bb6d0b7fc7bd98c3b Oct 30 05:57:04.545602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:04.545580 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:05.129229 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:05.129191 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerStarted","Data":"1ccb87b2408435229a2392cd46131c57c99d465b9b71f2968fb0cad685dacec7"} Oct 30 05:57:05.129662 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:05.129277 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerStarted","Data":"2181a54b9452d594ecfb02fece8dacae01a4175679fd633bb6d0b7fc7bd98c3b"} Oct 30 05:57:06.140903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:06.140864 6171 generic.go:356] "Generic (PLEG): container finished" podID="a8347028-f288-4b54-9174-d30374825a2f" containerID="1ccb87b2408435229a2392cd46131c57c99d465b9b71f2968fb0cad685dacec7" exitCode=7 Oct 30 05:57:06.141361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:06.140960 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerDied","Data":"1ccb87b2408435229a2392cd46131c57c99d465b9b71f2968fb0cad685dacec7"} Oct 30 05:57:06.141361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:06.141236 6171 scope.go:122] "RemoveContainer" containerID="1ccb87b2408435229a2392cd46131c57c99d465b9b71f2968fb0cad685dacec7" Oct 30 05:57:07.155276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.155237 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerStarted","Data":"2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3"} Oct 30 05:57:07.225326 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.225275 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" podStartSLOduration=4.225262452 podStartE2EDuration="4.225262452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:05.20188883 +0000 UTC m=+1483.642266025" watchObservedRunningTime="2026-10-30 05:57:07.225262452 +0000 UTC m=+1485.665639645" Oct 30 05:57:07.287790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.287755 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:57:07.288013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.287973 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" containerID="cri-o://77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90" gracePeriod=30 Oct 30 05:57:07.293431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.293400 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:57:07.293646 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.293621 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" containerID="cri-o://b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27" gracePeriod=30 Oct 30 05:57:07.299431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.299407 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:57:07.299580 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.299563 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" containerID="cri-o://daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66" gracePeriod=30 Oct 30 05:57:07.546565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.546531 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:57:07.546775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.546750 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" containerID="cri-o://fd25a347d056784cfaa55fb9a881667649b07719225fd77c9d32247a86b22906" gracePeriod=30 Oct 30 05:57:07.553092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.553026 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:57:07.553268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.553248 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" containerID="cri-o://8d6d9683bb042592d53ec32edc4064b8d9587ec68f53e0460338d1dcc3b11e82" gracePeriod=30 Oct 30 05:57:07.560454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.560424 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:57:07.560667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.560644 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" containerID="cri-o://a71d904634d3b3d8fa8091b5f29f0a9dabc76af4264ae4e8b2e8842fe8c164f3" gracePeriod=30 Oct 30 05:57:07.569290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.569268 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:57:07.569488 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:07.569467 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" containerID="cri-o://fc738844dd2fd83640952652ab5a3d8135e40883cdd48b6fa81fc825f38ebacd" gracePeriod=30 Oct 30 05:57:08.167917 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.167883 6171 generic.go:356] "Generic (PLEG): container finished" podID="a8347028-f288-4b54-9174-d30374825a2f" containerID="2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3" exitCode=7 Oct 30 05:57:08.168394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.167965 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerDied","Data":"2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3"} Oct 30 05:57:08.168394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.168066 6171 scope.go:122] "RemoveContainer" containerID="1ccb87b2408435229a2392cd46131c57c99d465b9b71f2968fb0cad685dacec7" Oct 30 05:57:08.168394 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.168226 6171 scope.go:122] "RemoveContainer" containerID="2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3" Oct 30 05:57:08.168594 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:08.168573 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5_odh-ai-gateway-infra(a8347028-f288-4b54-9174-d30374825a2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" podUID="a8347028-f288-4b54-9174-d30374825a2f" Oct 30 05:57:08.616068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.609980 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:57:08.616068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:08.610035 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:57:09.081552 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:09.081515 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:57:09.081724 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:09.081581 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:57:09.247147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:09.247118 6171 scope.go:122] "RemoveContainer" containerID="2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3" Oct 30 05:57:09.248323 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:09.248290 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5_odh-ai-gateway-infra(a8347028-f288-4b54-9174-d30374825a2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" podUID="a8347028-f288-4b54-9174-d30374825a2f" Oct 30 05:57:10.193998 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:10.193960 6171 patch_prober.go:29] interesting pod/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:57:10.194199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:10.194014 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:57:12.896157 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.884564 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:57:12.922308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.919337 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:57:12.929103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.922508 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994194 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994279 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994362 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994390 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994513 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994613 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994661 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994767 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994813 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994891 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994947 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.994984 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.995026 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.995074 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:12.997257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:12.995107 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:13.062067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.059194 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.080171 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.133905 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134182 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134616 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134670 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134699 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134864 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.134900 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135147 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135283 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135321 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135360 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135463 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\") pod \"0370a87b-2655-40d6-9c50-018d44049314\" (UID: \"0370a87b-2655-40d6-9c50-018d44049314\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135505 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\") pod \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\" (UID: \"30edac37-a8dd-40c5-bccd-53db5d8fd7cd\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.135538 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\") pod \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\" (UID: \"2afa0d1a-5836-4c90-8af6-5572f0979b9f\") " Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.136435 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.143069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.136766 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:13.170095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.169669 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.170095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.169848 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.170704 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.170363 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.191151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.186509 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192845 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192876 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192894 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192908 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192923 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192935 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192951 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192967 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192980 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.192993 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.193006 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.193421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.193021 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0370a87b-2655-40d6-9c50-018d44049314-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.221073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.212404 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.221073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.212742 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:13.221073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.217598 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.236078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.221969 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.267136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.265946 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.291141 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294270 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294567 6171 cpu_manager.go:414] "containerMap: removing container" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294591 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294662 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294670 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294682 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294690 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294908 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294929 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerName="istio-proxy" Oct 30 05:57:13.295101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.294939 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0370a87b-2655-40d6-9c50-018d44049314" containerName="istio-proxy" Oct 30 05:57:13.296459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296424 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.296459 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296458 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.296661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296471 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.296661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296483 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.296661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296495 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.296661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.296508 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.527296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.516519 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 05:57:13.527296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.516712 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.565514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.565333 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "kube-api-access-dsmlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.577090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.577016 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:13.620820 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.620774 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.627279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.627207 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.633649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.631942 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy" pod "2afa0d1a-5836-4c90-8af6-5572f0979b9f" (UID: "2afa0d1a-5836-4c90-8af6-5572f0979b9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:13.633649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.632033 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:13.636879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.634693 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "kube-api-access-nkx4l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638423 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638527 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-427hg\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638593 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638628 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638673 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638725 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638849 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.638981 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639278 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639465 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkx4l\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-kube-api-access-nkx4l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639483 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0370a87b-2655-40d6-9c50-018d44049314-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639498 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dsmlx\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-kube-api-access-dsmlx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639535 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639549 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2afa0d1a-5836-4c90-8af6-5572f0979b9f-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639563 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0370a87b-2655-40d6-9c50-018d44049314-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.656997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.639576 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.667172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.665988 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8" pod "30edac37-a8dd-40c5-bccd-53db5d8fd7cd" (UID: "30edac37-a8dd-40c5-bccd-53db5d8fd7cd"). InnerVolumeSpecName "kube-api-access-htdl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.755773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.755687 6171 generic.go:356] "Generic (PLEG): container finished" podID="2cca739a-b888-416c-98fc-64457cb99655" containerID="a71d904634d3b3d8fa8091b5f29f0a9dabc76af4264ae4e8b2e8842fe8c164f3" exitCode=0 Oct 30 05:57:13.769095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.767622 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.769095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.767708 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.769095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.767767 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-427hg\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.769095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.768872 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770492 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770548 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770589 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770620 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770712 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770793 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.772945 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.770962 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htdl8\" (UniqueName: \"kubernetes.io/projected/30edac37-a8dd-40c5-bccd-53db5d8fd7cd-kube-api-access-htdl8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.774748 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.776683 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.777252 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.777452 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.777893 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.778193 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.805080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.788111 6171 generic.go:356] "Generic (PLEG): container finished" podID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" containerID="b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27" exitCode=0 Oct 30 05:57:13.805080 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.788446 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" Oct 30 05:57:13.856081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.847430 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" Oct 30 05:57:13.856081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.847907 6171 generic.go:356] "Generic (PLEG): container finished" podID="0370a87b-2655-40d6-9c50-018d44049314" containerID="daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66" exitCode=0 Oct 30 05:57:13.923460 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.923423 6171 generic.go:356] "Generic (PLEG): container finished" podID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" containerID="77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90" exitCode=0 Oct 30 05:57:13.924625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.923931 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token" pod "0370a87b-2655-40d6-9c50-018d44049314" (UID: "0370a87b-2655-40d6-9c50-018d44049314"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.924752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.924544 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" Oct 30 05:57:13.929492 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.929470 6171 generic.go:356] "Generic (PLEG): container finished" podID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerID="fc738844dd2fd83640952652ab5a3d8135e40883cdd48b6fa81fc825f38ebacd" exitCode=0 Oct 30 05:57:13.932170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.932143 6171 generic.go:356] "Generic (PLEG): container finished" podID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerID="fd25a347d056784cfaa55fb9a881667649b07719225fd77c9d32247a86b22906" exitCode=0 Oct 30 05:57:13.935242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.935222 6171 generic.go:356] "Generic (PLEG): container finished" podID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerID="8d6d9683bb042592d53ec32edc4064b8d9587ec68f53e0460338d1dcc3b11e82" exitCode=0 Oct 30 05:57:13.942769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.942751 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:57:13.942987 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.942974 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:57:13.947497 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.947471 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" event={"ID":"2cca739a-b888-416c-98fc-64457cb99655","Type":"ContainerDied","Data":"a71d904634d3b3d8fa8091b5f29f0a9dabc76af4264ae4e8b2e8842fe8c164f3"} Oct 30 05:57:13.947659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.947645 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 05:57:13.953922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.953903 6171 scope.go:122] "RemoveContainer" containerID="a71d904634d3b3d8fa8091b5f29f0a9dabc76af4264ae4e8b2e8842fe8c164f3" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.955233 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.955761 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.955856 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.955873 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.956551 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" containerName="istio-proxy" Oct 30 05:57:13.957096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.956596 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cca739a-b888-416c-98fc-64457cb99655" containerName="istio-proxy" Oct 30 05:57:13.964334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.964318 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 05:57:13.964543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.964526 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" event={"ID":"30edac37-a8dd-40c5-bccd-53db5d8fd7cd","Type":"ContainerDied","Data":"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27"} Oct 30 05:57:13.964632 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.964621 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:57:13.967211 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.967194 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:13.980485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.980447 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-427hg\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.982522 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.982495 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\") pod \"e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:13.982745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.982689 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t" event={"ID":"30edac37-a8dd-40c5-bccd-53db5d8fd7cd","Type":"ContainerDied","Data":"dd80a42c840c39a282a44239976e17c1d4c7d55ed0da04f64f5c69bfac9889df"} Oct 30 05:57:13.982869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.982853 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" event={"ID":"0370a87b-2655-40d6-9c50-018d44049314","Type":"ContainerDied","Data":"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66"} Oct 30 05:57:13.982959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.982927 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:13.983187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.983162 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st" event={"ID":"0370a87b-2655-40d6-9c50-018d44049314","Type":"ContainerDied","Data":"4999bb95125fc7d719063b0fca6b8c39a575ff6ffd151a2558ed53abbc96be90"} Oct 30 05:57:13.983276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.983216 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" event={"ID":"2afa0d1a-5836-4c90-8af6-5572f0979b9f","Type":"ContainerDied","Data":"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90"} Oct 30 05:57:13.983276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.983251 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh" event={"ID":"2afa0d1a-5836-4c90-8af6-5572f0979b9f","Type":"ContainerDied","Data":"1a277da59d0b19bb9dd3cd3d402d6f76a90c52d5fde9082ab9f215dddbe64d0f"} Oct 30 05:57:13.984152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.983435 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerDied","Data":"fc738844dd2fd83640952652ab5a3d8135e40883cdd48b6fa81fc825f38ebacd"} Oct 30 05:57:13.985205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.985183 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" event={"ID":"e7453981-f89e-40b2-8190-105df4cd5a4b","Type":"ContainerDied","Data":"fd25a347d056784cfaa55fb9a881667649b07719225fd77c9d32247a86b22906"} Oct 30 05:57:13.985350 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.985333 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" event={"ID":"aff63842-10f8-4757-9b7f-7b8d89c49e5b","Type":"ContainerDied","Data":"8d6d9683bb042592d53ec32edc4064b8d9587ec68f53e0460338d1dcc3b11e82"} Oct 30 05:57:13.988722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.988576 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:57:13.994893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:13.994831 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-cc33d3-serving-cert\"" Oct 30 05:57:14.006456 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.006396 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz"] Oct 30 05:57:14.008526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.008154 6171 scope.go:122] "RemoveContainer" containerID="b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27" Oct 30 05:57:14.016373 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.016352 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-1b2243ef-4mzdz"] Oct 30 05:57:14.023750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023567 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.023750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023631 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.023750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023679 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.023750 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023740 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023806 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023856 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023937 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024013 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.023980 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024024 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024087 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024117 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024209 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024302 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.024673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024481 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024519 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024579 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\") pod \"e7453981-f89e-40b2-8190-105df4cd5a4b\" (UID: \"e7453981-f89e-40b2-8190-105df4cd5a4b\") " Oct 30 05:57:14.024924 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.024689 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\") pod \"2cca739a-b888-416c-98fc-64457cb99655\" (UID: \"2cca739a-b888-416c-98fc-64457cb99655\") " Oct 30 05:57:14.025028 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025005 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.025136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025078 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.025366 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.025429 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025361 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhbtn\" (UniqueName: \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.025491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025465 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.025554 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025507 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.025764 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025745 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.025838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025823 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhtwg\" (UniqueName: \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.025942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025930 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0370a87b-2655-40d6-9c50-018d44049314-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025014 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.025297 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.026127 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.026397 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.026822 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.027008 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.029404 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.029600 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.029839 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:14.030018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.029934 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.030161 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.030651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.030519 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.031096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.030986 6171 scope.go:122] "RemoveContainer" containerID="b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27" Oct 30 05:57:14.033137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.033110 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "kube-api-access-gh5x2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.033230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.033193 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "kube-api-access-wzfx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.034819 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:14.034790 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27\": container with ID starting with b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27 not found: ID does not exist" containerID="b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27" Oct 30 05:57:14.034919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.034853 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27"} err="failed to get container status \"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27\": rpc error: code = NotFound desc = could not find container \"b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27\": container with ID starting with b1555fb11dd04434c2491383ba0a4b2c548e1fe97a8a6a54ba46b6bef926fa27 not found: ID does not exist" Oct 30 05:57:14.034919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.034876 6171 scope.go:122] "RemoveContainer" containerID="daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66" Oct 30 05:57:14.040262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.040227 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:14.040389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.040362 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.042577 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.042491 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy" pod "e7453981-f89e-40b2-8190-105df4cd5a4b" (UID: "e7453981-f89e-40b2-8190-105df4cd5a4b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.048569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.048445 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:57:14.054948 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.054933 6171 scope.go:122] "RemoveContainer" containerID="daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66" Oct 30 05:57:14.055222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.055197 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token" pod "2cca739a-b888-416c-98fc-64457cb99655" (UID: "2cca739a-b888-416c-98fc-64457cb99655"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.055398 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:14.055372 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66\": container with ID starting with daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66 not found: ID does not exist" containerID="daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66" Oct 30 05:57:14.055474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.055410 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66"} err="failed to get container status \"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66\": rpc error: code = NotFound desc = could not find container \"daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66\": container with ID starting with daab8dc949eab6ffb03ba4b2e7f86dc21134b6b37aff5ee43e837c778d271d66 not found: ID does not exist" Oct 30 05:57:14.055474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.055434 6171 scope.go:122] "RemoveContainer" containerID="77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90" Oct 30 05:57:14.072504 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.072488 6171 scope.go:122] "RemoveContainer" containerID="77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90" Oct 30 05:57:14.073416 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:14.073386 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90\": container with ID starting with 77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90 not found: ID does not exist" containerID="77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90" Oct 30 05:57:14.073528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.073428 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90"} err="failed to get container status \"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90\": rpc error: code = NotFound desc = could not find container \"77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90\": container with ID starting with 77677a9f2b6c7173a31e905a31dd95770918d3a051efc497125dd2e7b4859f90 not found: ID does not exist" Oct 30 05:57:14.073528 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.073452 6171 scope.go:122] "RemoveContainer" containerID="223228bd9680e59c6a9eb9831156a259b57da898246a1bad4823232487811608" Oct 30 05:57:14.085716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.085695 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:14.086213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.086190 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:57:14.086421 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.086405 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-3571f8-serving-cert\"" Oct 30 05:57:14.090976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.090953 6171 scope.go:122] "RemoveContainer" containerID="fd25a347d056784cfaa55fb9a881667649b07719225fd77c9d32247a86b22906" Oct 30 05:57:14.112730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.112703 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:57:14.113953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.113931 6171 cpu_manager.go:414] "containerMap: removing container" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" Oct 30 05:57:14.113953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.113955 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" Oct 30 05:57:14.114163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.113986 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:14.114163 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.113994 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:14.114398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.114378 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:14.114446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.114425 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" containerName="istio-proxy" Oct 30 05:57:14.126565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126545 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.126773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126746 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.126862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126796 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.126929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126886 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.126994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126944 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.126994 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.126984 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.127284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.127103 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.127284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.127142 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.127284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.127178 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.127284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.127276 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.127480 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.127315 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.128897 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.128935 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129022 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129033 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129192 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129248 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129277 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129289 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129363 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129408 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\") pod \"ec17335d-d8f7-40f6-a3ef-646b49be1384\" (UID: \"ec17335d-d8f7-40f6-a3ef-646b49be1384\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129441 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129466 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\") pod \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\" (UID: \"aff63842-10f8-4757-9b7f-7b8d89c49e5b\") " Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129701 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129819 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129860 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.129934 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130200 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhbtn\" (UniqueName: \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130299 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130345 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130585 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130587 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130677 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhtwg\" (UniqueName: \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130844 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130862 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzfx2\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-kube-api-access-wzfx2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130877 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh5x2\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-kube-api-access-gh5x2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130892 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130905 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130918 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130932 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130946 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130962 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2cca739a-b888-416c-98fc-64457cb99655-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130976 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.130990 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131007 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131022 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131038 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2cca739a-b888-416c-98fc-64457cb99655-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131075 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7453981-f89e-40b2-8190-105df4cd5a4b-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131090 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7453981-f89e-40b2-8190-105df4cd5a4b-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131105 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131118 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131131 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2cca739a-b888-416c-98fc-64457cb99655-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131145 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7453981-f89e-40b2-8190-105df4cd5a4b-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131158 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131172 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131186 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131198 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2cca739a-b888-416c-98fc-64457cb99655-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131200 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.132307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.131491 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:14.134610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.134581 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.134864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.134768 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.138415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.138132 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.138415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.138250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:14.141813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.141792 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.141813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.141793 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.142136 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.142065 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.142450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.142414 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:14.142639 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.142599 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "kube-api-access-w9xt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.142919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.142863 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "kube-api-access-jcmmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.143485 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.143417 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:14.145000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.144611 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:14.145000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.144820 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token" pod "aff63842-10f8-4757-9b7f-7b8d89c49e5b" (UID: "aff63842-10f8-4757-9b7f-7b8d89c49e5b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.145512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.145068 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.146348 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.146173 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token" pod "ec17335d-d8f7-40f6-a3ef-646b49be1384" (UID: "ec17335d-d8f7-40f6-a3ef-646b49be1384"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:14.160733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.160700 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:57:14.187452 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.187417 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d4ccc5-serving-cert\"" Oct 30 05:57:14.187825 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.187797 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhtwg\" (UniqueName: \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\") pod \"maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.192988 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.192926 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:57:14.193286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.193265 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhbtn\" (UniqueName: \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\") pod \"maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.200881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.200621 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-mrd4t"] Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247070 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zptc\" (UniqueName: \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247200 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247300 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247537 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247695 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247714 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247726 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec17335d-d8f7-40f6-a3ef-646b49be1384-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247741 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcmmp\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-kube-api-access-jcmmp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247753 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247766 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247777 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247789 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247800 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec17335d-d8f7-40f6-a3ef-646b49be1384-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247811 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247823 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9xt6\" (UniqueName: \"kubernetes.io/projected/aff63842-10f8-4757-9b7f-7b8d89c49e5b-kube-api-access-w9xt6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.248603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.247836 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aff63842-10f8-4757-9b7f-7b8d89c49e5b-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:14.302752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.302671 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:57:14.310629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.310601 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-5nnvh"] Oct 30 05:57:14.329923 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.329898 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:14.349754 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.349731 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.349873 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.349834 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zptc\" (UniqueName: \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.349940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.349911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.350005 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.349974 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.350500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.350476 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.352510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.352492 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.352939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.352921 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.383539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.365954 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:14.388932 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.388895 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zptc\" (UniqueName: \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\") pod \"maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.467406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.460240 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:14.497187 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.497091 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 05:57:14.513483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.512399 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:14.513483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.512501 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:14.513483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.512550 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:14.554961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.554900 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:57:14.563755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.563728 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-l58st"] Oct 30 05:57:14.742527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.742491 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 05:57:14.743946 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:14.743916 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e07cc31_169a_4507_8d3b_ff4476eb4519.slice/crio-f9e3827fc51a919b7a2cac3735ca17e59fe0903545979e84371d7e2739960b6a WatchSource:0}: Error finding container f9e3827fc51a919b7a2cac3735ca17e59fe0903545979e84371d7e2739960b6a: Status 404 returned error can't find the container with id f9e3827fc51a919b7a2cac3735ca17e59fe0903545979e84371d7e2739960b6a Oct 30 05:57:14.745581 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:14.745558 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ad9a498_4870_4fd8_955b_d8af89bdeb97.slice/crio-a1e7b97c4b9af306f0ea766edbb847fc51526502382c09ae605e97855e6b7f2f WatchSource:0}: Error finding container a1e7b97c4b9af306f0ea766edbb847fc51526502382c09ae605e97855e6b7f2f: Status 404 returned error can't find the container with id a1e7b97c4b9af306f0ea766edbb847fc51526502382c09ae605e97855e6b7f2f Oct 30 05:57:14.750143 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.750121 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:57:14.810511 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:14.810475 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13896cfb_594f_4652_ac3e_6dc1940da08b.slice/crio-0fc1d435e6e9590f4861fed658e6991c4dcfe830d57750250ef8e4a51dea63f5 WatchSource:0}: Error finding container 0fc1d435e6e9590f4861fed658e6991c4dcfe830d57750250ef8e4a51dea63f5: Status 404 returned error can't find the container with id 0fc1d435e6e9590f4861fed658e6991c4dcfe830d57750250ef8e4a51dea63f5 Oct 30 05:57:14.811510 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.811481 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:57:14.947365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.947336 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" Oct 30 05:57:14.947365 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.947348 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982" event={"ID":"aff63842-10f8-4757-9b7f-7b8d89c49e5b","Type":"ContainerDied","Data":"5bba0503e1c1cc02b3ae8095402b546185347ff4c47ddfdf7e281b114fea802f"} Oct 30 05:57:14.947822 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.947414 6171 scope.go:122] "RemoveContainer" containerID="8d6d9683bb042592d53ec32edc4064b8d9587ec68f53e0460338d1dcc3b11e82" Oct 30 05:57:14.950695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.949778 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" event={"ID":"63894899-4069-4607-824b-71d0ddaa304e","Type":"ContainerStarted","Data":"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957"} Oct 30 05:57:14.950695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.949825 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" event={"ID":"63894899-4069-4607-824b-71d0ddaa304e","Type":"ContainerStarted","Data":"8f213ccf7e11aa92548f6c6a1804d46a78ad6e06e24a1a2acf938595a7d87ce3"} Oct 30 05:57:14.951382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.951361 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" event={"ID":"1ad9a498-4870-4fd8-955b-d8af89bdeb97","Type":"ContainerStarted","Data":"a1e7b97c4b9af306f0ea766edbb847fc51526502382c09ae605e97855e6b7f2f"} Oct 30 05:57:14.952879 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.952854 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" event={"ID":"13896cfb-594f-4652-ac3e-6dc1940da08b","Type":"ContainerStarted","Data":"0fc1d435e6e9590f4861fed658e6991c4dcfe830d57750250ef8e4a51dea63f5"} Oct 30 05:57:14.954309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.954286 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" event={"ID":"2e07cc31-169a-4507-8d3b-ff4476eb4519","Type":"ContainerStarted","Data":"f9e3827fc51a919b7a2cac3735ca17e59fe0903545979e84371d7e2739960b6a"} Oct 30 05:57:14.955418 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.955392 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" Oct 30 05:57:14.955517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.955444 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp" event={"ID":"2cca739a-b888-416c-98fc-64457cb99655","Type":"ContainerDied","Data":"1e0df3be34aab6f4e92a9fc0705dfddc499c5b6020889d64f5689c083da213b9"} Oct 30 05:57:14.964091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.961873 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" Oct 30 05:57:14.964091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.961932 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw" event={"ID":"ec17335d-d8f7-40f6-a3ef-646b49be1384","Type":"ContainerDied","Data":"ad72806739a87945facfdee3113d152a21f551c07b11733effb5e88be6e52411"} Oct 30 05:57:14.964091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.962994 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" Oct 30 05:57:14.964091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.963024 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv" event={"ID":"e7453981-f89e-40b2-8190-105df4cd5a4b","Type":"ContainerDied","Data":"db303689fc3776f9a10d23f8a542da6bc96ae05fddccde49bbb23426073e793f"} Oct 30 05:57:14.964091 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:14.963684 6171 scope.go:122] "RemoveContainer" containerID="fc738844dd2fd83640952652ab5a3d8135e40883cdd48b6fa81fc825f38ebacd" Oct 30 05:57:15.070454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.070359 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podStartSLOduration=41.070344022 podStartE2EDuration="41.070344022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:15.067281958 +0000 UTC m=+1493.507659152" watchObservedRunningTime="2026-10-30 05:57:15.070344022 +0000 UTC m=+1493.510721212" Oct 30 05:57:15.086451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.086420 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:15.087679 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.087649 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:15.087805 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.087700 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:15.169100 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.169073 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:57:15.179252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.179222 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-df982"] Oct 30 05:57:15.233096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.233027 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:57:15.238250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.238221 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-n9cxw"] Oct 30 05:57:15.473070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.464873 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:57:15.529068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.528599 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-mjhrv"] Oct 30 05:57:15.544074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.543992 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:57:15.560116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.553922 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-a35835-openshift-default-56c7cfb9d5-8gwlp"] Oct 30 05:57:15.676956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.676885 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0370a87b-2655-40d6-9c50-018d44049314" path="/var/lib/kubelet/pods/0370a87b-2655-40d6-9c50-018d44049314/volumes" Oct 30 05:57:15.677407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.677392 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" path="/var/lib/kubelet/pods/2afa0d1a-5836-4c90-8af6-5572f0979b9f/volumes" Oct 30 05:57:15.677850 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.677832 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0de957-a41f-4038-8f6a-8e86e4720080" path="/var/lib/kubelet/pods/2c0de957-a41f-4038-8f6a-8e86e4720080/volumes" Oct 30 05:57:15.678696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.678678 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cca739a-b888-416c-98fc-64457cb99655" path="/var/lib/kubelet/pods/2cca739a-b888-416c-98fc-64457cb99655/volumes" Oct 30 05:57:15.679174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.679160 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30edac37-a8dd-40c5-bccd-53db5d8fd7cd" path="/var/lib/kubelet/pods/30edac37-a8dd-40c5-bccd-53db5d8fd7cd/volumes" Oct 30 05:57:15.679626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.679613 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff63842-10f8-4757-9b7f-7b8d89c49e5b" path="/var/lib/kubelet/pods/aff63842-10f8-4757-9b7f-7b8d89c49e5b/volumes" Oct 30 05:57:15.694070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.680499 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7453981-f89e-40b2-8190-105df4cd5a4b" path="/var/lib/kubelet/pods/e7453981-f89e-40b2-8190-105df4cd5a4b/volumes" Oct 30 05:57:15.702109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:15.695157 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" path="/var/lib/kubelet/pods/ec17335d-d8f7-40f6-a3ef-646b49be1384/volumes" Oct 30 05:57:16.059777 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.059728 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" event={"ID":"1ad9a498-4870-4fd8-955b-d8af89bdeb97","Type":"ContainerStarted","Data":"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d"} Oct 30 05:57:16.063525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.063497 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" event={"ID":"13896cfb-594f-4652-ac3e-6dc1940da08b","Type":"ContainerStarted","Data":"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de"} Oct 30 05:57:16.063695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.063549 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:16.066563 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.066538 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" event={"ID":"2e07cc31-169a-4507-8d3b-ff4476eb4519","Type":"ContainerStarted","Data":"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf"} Oct 30 05:57:16.067174 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.067150 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:16.106081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.088427 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:16.106081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.088477 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:16.147949 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.147894 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" podStartSLOduration=91.800326392 podStartE2EDuration="1m32.147880217s" totalImagesPullingTime="347.553825ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:16.144156388 +0000 UTC m=+1494.584533572" watchObservedRunningTime="2026-10-30 05:57:16.147880217 +0000 UTC m=+1494.588257410" Oct 30 05:57:16.213773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.213676 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" podStartSLOduration=91.865206115 podStartE2EDuration="1m32.213656445s" totalImagesPullingTime="348.45033ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:16.211391461 +0000 UTC m=+1494.651768675" watchObservedRunningTime="2026-10-30 05:57:16.213656445 +0000 UTC m=+1494.654033638" Oct 30 05:57:16.270004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.269949 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" podStartSLOduration=91.911918962 podStartE2EDuration="1m32.269935576s" totalImagesPullingTime="358.016614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:16.268926577 +0000 UTC m=+1494.709303769" watchObservedRunningTime="2026-10-30 05:57:16.269935576 +0000 UTC m=+1494.710312769" Oct 30 05:57:16.431612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.430108 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 05:57:16.431612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.431151 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:16.431612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.431170 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:16.431612 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.431443 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec17335d-d8f7-40f6-a3ef-646b49be1384" containerName="istio-proxy" Oct 30 05:57:16.440019 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.439994 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 05:57:16.440220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.440103 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.591069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.579563 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-de7242-openshift-default-dockercfg-x5rjm\"" Oct 30 05:57:16.655068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.592988 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.655068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.593174 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.655068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.593282 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.655068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.593348 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.733827 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.733787 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.764292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.764241 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljjw6\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.764629 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.764608 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.764888 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.764861 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.765209 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.765176 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889318 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889453 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889517 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889548 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889602 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljjw6\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889664 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889760 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889831 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.889871 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.891697 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.892938 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.894477 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.894733 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.909320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.909290 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.936559 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.936525 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.949086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.937488 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljjw6\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.953449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.953378 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:16.981086 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:16.971422 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\") pod \"e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:17.060495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.060410 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:17.078647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.078535 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:17.100037 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.099998 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:17.100233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.100078 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:17.537203 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:17.537170 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41cf7f40_7748_4be6_bedc_be1c7e3b7b1d.slice/crio-16a030fef44d38f32a0b855a76d7d1767480cb8dd5d8be5fd5209640db12cb2e WatchSource:0}: Error finding container 16a030fef44d38f32a0b855a76d7d1767480cb8dd5d8be5fd5209640db12cb2e: Status 404 returned error can't find the container with id 16a030fef44d38f32a0b855a76d7d1767480cb8dd5d8be5fd5209640db12cb2e Oct 30 05:57:17.539110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.539084 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 05:57:17.540538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.540499 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:17.540622 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.540569 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:17.540622 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:17.540598 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:18.099101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:18.098271 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:18.099101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:18.098344 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:18.148069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:18.146694 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerStarted","Data":"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83"} Oct 30 05:57:18.148069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:18.146739 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerStarted","Data":"16a030fef44d38f32a0b855a76d7d1767480cb8dd5d8be5fd5209640db12cb2e"} Oct 30 05:57:18.230068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:18.229617 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podStartSLOduration=2.229584962 podStartE2EDuration="2.229584962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:18.211707633 +0000 UTC m=+1496.652084866" watchObservedRunningTime="2026-10-30 05:57:18.229584962 +0000 UTC m=+1496.669962156" Oct 30 05:57:19.061621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.061583 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:19.067207 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.067181 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:19.106680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.106630 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:19.107193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.106697 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:19.156118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.156089 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:19.156902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.156880 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:57:19.667826 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:19.667797 6171 scope.go:122] "RemoveContainer" containerID="2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3" Oct 30 05:57:20.105068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:20.101418 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:20.105068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:20.101471 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:21.089190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.088449 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body= Oct 30 05:57:21.089190 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.088513 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" Oct 30 05:57:21.191824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.191732 6171 generic.go:356] "Generic (PLEG): container finished" podID="a8347028-f288-4b54-9174-d30374825a2f" containerID="367d77f3fd3cc27a343a936e124e2dc82f0b98b875d5be22c9a6d35a8e36d13d" exitCode=7 Oct 30 05:57:21.191824 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.191808 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerDied","Data":"367d77f3fd3cc27a343a936e124e2dc82f0b98b875d5be22c9a6d35a8e36d13d"} Oct 30 05:57:21.192076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.191885 6171 scope.go:122] "RemoveContainer" containerID="2aac0bed65f8d607d91f4d59c6499537fbabd3cc061b59cd2fb6bb1f8214e9d3" Oct 30 05:57:21.192518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:21.192496 6171 scope.go:122] "RemoveContainer" containerID="367d77f3fd3cc27a343a936e124e2dc82f0b98b875d5be22c9a6d35a8e36d13d" Oct 30 05:57:21.194081 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:21.193020 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5_odh-ai-gateway-infra(a8347028-f288-4b54-9174-d30374825a2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" podUID="a8347028-f288-4b54-9174-d30374825a2f" Oct 30 05:57:22.084393 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.084360 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:57:22.090593 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.090568 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:22.090933 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.090831 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:22.091647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.091626 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 05:57:22.256505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.256465 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:22.494160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.494123 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:22.543893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.543866 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\" (UniqueName: \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\") pod \"a8347028-f288-4b54-9174-d30374825a2f\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " Oct 30 05:57:22.544111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.543931 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\") pod \"a8347028-f288-4b54-9174-d30374825a2f\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " Oct 30 05:57:22.544111 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.544016 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\") pod \"a8347028-f288-4b54-9174-d30374825a2f\" (UID: \"a8347028-f288-4b54-9174-d30374825a2f\") " Oct 30 05:57:22.544286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.544237 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca" pod "a8347028-f288-4b54-9174-d30374825a2f" (UID: "a8347028-f288-4b54-9174-d30374825a2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:22.544784 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.544763 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8347028-f288-4b54-9174-d30374825a2f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:22.546020 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.545997 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:57:22.546253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.546229 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2" pod "a8347028-f288-4b54-9174-d30374825a2f" (UID: "a8347028-f288-4b54-9174-d30374825a2f"). InnerVolumeSpecName "kube-api-access-2vxm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:22.546324 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.546227 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca" pod "a8347028-f288-4b54-9174-d30374825a2f" (UID: "a8347028-f288-4b54-9174-d30374825a2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:57:22.547178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547157 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.547178 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547177 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.547329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547235 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.547329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547244 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.547495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547480 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.547549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.547499 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.551638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.551620 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.626116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.620491 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1b2243ef-dockercfg-nqzpf\"" Oct 30 05:57:22.626116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.620587 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1b2243ef\"" Oct 30 05:57:22.671071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.661562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.671071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.661619 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5fmd\" (UniqueName: \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.671071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.662015 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8347028-f288-4b54-9174-d30374825a2f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:22.671071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.662035 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vxm2\" (UniqueName: \"kubernetes.io/projected/a8347028-f288-4b54-9174-d30374825a2f-kube-api-access-2vxm2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:22.697252 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.696099 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:57:22.802074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.796470 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.802074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.796622 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5fmd\" (UniqueName: \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.812506 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.812473 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.875499 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.869664 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5fmd\" (UniqueName: \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\") pod \"payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:22.911131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.902271 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:57:22.911131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.903782 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.911131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.903813 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.935169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.924306 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8347028-f288-4b54-9174-d30374825a2f" containerName="revoke-keys" Oct 30 05:57:22.991301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.991264 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:57:22.991633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:22.991614 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.117803 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.117769 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:23.119399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.119370 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.119542 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.119479 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9bs9\" (UniqueName: \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.153871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.153840 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 05:57:23.153992 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.153909 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:57:23.227649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.226482 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.227649 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.226653 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9bs9\" (UniqueName: \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.237708 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.237677 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.257836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.257799 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" event={"ID":"a8347028-f288-4b54-9174-d30374825a2f","Type":"ContainerDied","Data":"2181a54b9452d594ecfb02fece8dacae01a4175679fd633bb6d0b7fc7bd98c3b"} Oct 30 05:57:23.258074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.258057 6171 scope.go:122] "RemoveContainer" containerID="367d77f3fd3cc27a343a936e124e2dc82f0b98b875d5be22c9a6d35a8e36d13d" Oct 30 05:57:23.258251 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.257843 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5" Oct 30 05:57:23.268789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.268736 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9bs9\" (UniqueName: \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\") pod \"payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.330284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.330259 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:23.337197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.337171 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-vj4g5"] Oct 30 05:57:23.425727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.425700 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:23.447650 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:23.447622 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730b950b_5e83_4b84_b92b_ebb3f45560af.slice/crio-17d3e8a9f02cdb21eaa9ccf06b62f8cee7abd0c98d8335252ce6c4b9a688b60b WatchSource:0}: Error finding container 17d3e8a9f02cdb21eaa9ccf06b62f8cee7abd0c98d8335252ce6c4b9a688b60b: Status 404 returned error can't find the container with id 17d3e8a9f02cdb21eaa9ccf06b62f8cee7abd0c98d8335252ce6c4b9a688b60b Oct 30 05:57:23.449602 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.449577 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:57:23.693829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.693766 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8347028-f288-4b54-9174-d30374825a2f" path="/var/lib/kubelet/pods/a8347028-f288-4b54-9174-d30374825a2f/volumes" Oct 30 05:57:23.948785 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:23.948750 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9900f8_0bd5_4e87_9947_99575bf2e077.slice/crio-0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26 WatchSource:0}: Error finding container 0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26: Status 404 returned error can't find the container with id 0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26 Oct 30 05:57:23.948990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:23.948975 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:57:24.274056 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.274000 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" event={"ID":"730b950b-5e83-4b84-b92b-ebb3f45560af","Type":"ContainerStarted","Data":"8dd5394836ed3dc1e90ef821a5aa8e1e22e94a105e12cdc934cdfda1e6772b20"} Oct 30 05:57:24.274518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.274097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:24.274518 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.274124 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" event={"ID":"730b950b-5e83-4b84-b92b-ebb3f45560af","Type":"ContainerStarted","Data":"17d3e8a9f02cdb21eaa9ccf06b62f8cee7abd0c98d8335252ce6c4b9a688b60b"} Oct 30 05:57:24.277770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.277744 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" event={"ID":"fa9900f8-0bd5-4e87-9947-99575bf2e077","Type":"ContainerStarted","Data":"2100670daaaf02e22db323cc82687b07c604f8eb7d804dc91602c4c9bff03535"} Oct 30 05:57:24.277926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.277789 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" event={"ID":"fa9900f8-0bd5-4e87-9947-99575bf2e077","Type":"ContainerStarted","Data":"0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26"} Oct 30 05:57:24.277926 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.277885 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:24.328104 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.328038 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" podStartSLOduration=2.3280238730000002 podStartE2EDuration="2.328023873s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:24.322146764 +0000 UTC m=+1502.762523957" watchObservedRunningTime="2026-10-30 05:57:24.328023873 +0000 UTC m=+1502.768401065" Oct 30 05:57:24.376428 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.376364 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" podStartSLOduration=2.376350322 podStartE2EDuration="2.376350322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:24.372996339 +0000 UTC m=+1502.813373533" watchObservedRunningTime="2026-10-30 05:57:24.376350322 +0000 UTC m=+1502.816727550" Oct 30 05:57:24.431216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.431177 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 05:57:24.436283 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.436264 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:24.456663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.456638 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-de7242\"" Oct 30 05:57:24.456823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.456665 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-de7242-dockercfg-xptkd\"" Oct 30 05:57:24.464300 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.464278 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 05:57:24.572128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.569977 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:24.572128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.570087 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5lsn\" (UniqueName: \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:24.726071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.708075 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:24.726071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.708166 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5lsn\" (UniqueName: \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:24.726071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.709651 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:25.004070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.999945 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5lsn\" (UniqueName: \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\") pod \"payload-processing-e2e-inf-b-de7242-777f988ccf-879xx\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:25.004070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:24.999996 6171 scope.go:122] "RemoveContainer" containerID="c14c8f4098cbbf3c5b5510b68195943ee9752dd8cb1e698888e5f5355c904bee" Oct 30 05:57:25.041551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.041528 6171 scope.go:122] "RemoveContainer" containerID="002c7b7f7177123374bcfec5ba3499fef3367c69d3c39b561c1126b9135b161d" Oct 30 05:57:25.053900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.053876 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:25.057767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.057751 6171 scope.go:122] "RemoveContainer" containerID="554733eb8a0415f4277cc4eada073c74cf75fe5dfd68ebb9833a22b32b910693" Oct 30 05:57:25.082094 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.082032 6171 scope.go:122] "RemoveContainer" containerID="698378bc9973ef82b9c5b583fad6f8d9db6d25873237a103f8b859710b2c62bc" Oct 30 05:57:25.123244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.123221 6171 scope.go:122] "RemoveContainer" containerID="2bafb2707c81f700c20eeb4f49c6f84cd5741c8834da0d73a2788e621e038abf" Oct 30 05:57:25.145549 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.145218 6171 scope.go:122] "RemoveContainer" containerID="a1e32948077ada63724672c8959c17fb2ee8359635ce22b54744d8de7cb939f3" Oct 30 05:57:25.159953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.159931 6171 scope.go:122] "RemoveContainer" containerID="5c20a0533a8c353b2ba9977f729b88f86a9482bf50124321ccbd5f558869a164" Oct 30 05:57:25.173655 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.173632 6171 scope.go:122] "RemoveContainer" containerID="23fa15c954f414e33eb0dc8000df884b275d60506770edb0c41329862f850276" Oct 30 05:57:25.186383 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.186364 6171 scope.go:122] "RemoveContainer" containerID="0a73ceb46af4cb7cf8f220dee7e7150439c2532ff2e580940b2123c836ba9c30" Oct 30 05:57:25.200223 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.200199 6171 scope.go:122] "RemoveContainer" containerID="892e7946ea2137258c733de4cdea250a22c44fb5aabf4b9966ce7045b2a38694" Oct 30 05:57:25.212598 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.212575 6171 scope.go:122] "RemoveContainer" containerID="ea50dfffc0af5b3bb858bcea7416576f24a217ebe02921c95a3eb851e451e465" Oct 30 05:57:25.223792 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.223776 6171 scope.go:122] "RemoveContainer" containerID="5c237f25166c77c8ecee58b644110c95ad9bb2ae37b557f7691f478a3d5355c8" Oct 30 05:57:25.233791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.233773 6171 scope.go:122] "RemoveContainer" containerID="761c48e4da7a7a500171054ce15e8d1832c8ecdf3d32092f4d64bb7dcc222fce" Oct 30 05:57:25.386024 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:25.385982 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83154825_18ec_4193_8aff_3983a8c47219.slice/crio-ff30cef189f2c9b9734a87c7321d1911a4317ca96f90e953e9279cc5a7a038e6 WatchSource:0}: Error finding container ff30cef189f2c9b9734a87c7321d1911a4317ca96f90e953e9279cc5a7a038e6: Status 404 returned error can't find the container with id ff30cef189f2c9b9734a87c7321d1911a4317ca96f90e953e9279cc5a7a038e6 Oct 30 05:57:25.386471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:25.386365 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 05:57:26.316255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.316217 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" event={"ID":"83154825-18ec-4193-8aff-3983a8c47219","Type":"ContainerStarted","Data":"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630"} Oct 30 05:57:26.316433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.316273 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" event={"ID":"83154825-18ec-4193-8aff-3983a8c47219","Type":"ContainerStarted","Data":"ff30cef189f2c9b9734a87c7321d1911a4317ca96f90e953e9279cc5a7a038e6"} Oct 30 05:57:26.316433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.316302 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:26.366338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.366288 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" podStartSLOduration=2.366274657 podStartE2EDuration="2.366274657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:26.361869097 +0000 UTC m=+1504.802246291" watchObservedRunningTime="2026-10-30 05:57:26.366274657 +0000 UTC m=+1504.806651851" Oct 30 05:57:26.384995 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:26.384957 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:57:26.715304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.715226 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b"] Oct 30 05:57:26.721151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.721128 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.737839 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.737814 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:57:26.741367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.741348 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b"] Oct 30 05:57:26.747239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.747210 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:57:26.820090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.820027 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.820246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.820221 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbvzx\" (UniqueName: \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.820298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.820286 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.922389 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.922359 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.922561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.922491 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbvzx\" (UniqueName: \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.922561 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.922535 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.922970 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.922951 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.925000 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.924979 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:26.948765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:26.948736 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbvzx\" (UniqueName: \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\") pod \"maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:27.038606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:27.038571 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:27.376389 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:27.376345 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3aa13d_2b73_40d8_a25f_0a03d87b2aca.slice/crio-0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba WatchSource:0}: Error finding container 0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba: Status 404 returned error can't find the container with id 0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba Oct 30 05:57:27.379077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:27.379031 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b"] Oct 30 05:57:28.366936 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:28.366902 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" event={"ID":"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca","Type":"ContainerStarted","Data":"f4b74026151511ae6c9559ce0adc30418ed16098f275457d969ff06f41d4af13"} Oct 30 05:57:28.367382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:28.366951 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" event={"ID":"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca","Type":"ContainerStarted","Data":"0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba"} Oct 30 05:57:28.416696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:28.416648 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" podStartSLOduration=2.416634867 podStartE2EDuration="2.416634867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:28.410712353 +0000 UTC m=+1506.851089548" watchObservedRunningTime="2026-10-30 05:57:28.416634867 +0000 UTC m=+1506.857012081" Oct 30 05:57:29.378823 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:29.378791 6171 generic.go:356] "Generic (PLEG): container finished" podID="3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" containerID="f4b74026151511ae6c9559ce0adc30418ed16098f275457d969ff06f41d4af13" exitCode=0 Oct 30 05:57:29.379269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:29.378877 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" event={"ID":"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca","Type":"ContainerDied","Data":"f4b74026151511ae6c9559ce0adc30418ed16098f275457d969ff06f41d4af13"} Oct 30 05:57:30.574011 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.573986 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:30.722199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.714716 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\") pod \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " Oct 30 05:57:30.722199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.715408 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca" pod "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" (UID: "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:30.722199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.715628 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\") pod \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " Oct 30 05:57:30.722199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.717351 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\" (UniqueName: \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\") pod \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\" (UID: \"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca\") " Oct 30 05:57:30.766062 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.766001 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:30.768618 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.768581 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca" pod "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" (UID: "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:57:30.768754 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.768737 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx" pod "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" (UID: "3a3aa13d-2b73-40d8-a25f-0a03d87b2aca"). InnerVolumeSpecName "kube-api-access-rbvzx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:30.867848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.867821 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:30.867848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:30.867844 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbvzx\" (UniqueName: \"kubernetes.io/projected/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca-kube-api-access-rbvzx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:31.398730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:31.398702 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" Oct 30 05:57:31.398730 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:31.398735 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b" event={"ID":"3a3aa13d-2b73-40d8-a25f-0a03d87b2aca","Type":"ContainerDied","Data":"0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba"} Oct 30 05:57:31.399026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:31.398783 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0dee56388d05d8fc646a990c1f762e0dd4a28195d9765e825c1f31ce1c4c62ba" Oct 30 05:57:35.308466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:35.308430 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:57:35.308907 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:35.308519 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:57:37.014294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.014246 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:57:37.014703 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.014471 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" containerID="cri-o://1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd" gracePeriod=30 Oct 30 05:57:37.028278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.028248 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:57:37.028524 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.028496 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" containerID="cri-o://b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea" gracePeriod=30 Oct 30 05:57:37.035793 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.035766 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:57:37.036197 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.036161 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" containerID="cri-o://86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945" gracePeriod=30 Oct 30 05:57:37.240405 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.240361 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:57:37.255491 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.255465 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:57:37.358707 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.358631 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 05:57:37.893334 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.893294 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:57:37.929202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:37.929162 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:57:38.118296 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:38.118256 6171 patch_prober.go:29] interesting pod/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:57:38.118842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:38.118811 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:57:40.197474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:40.197404 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:57:42.241101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.240930 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" podUID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" containerName="payload-processing" containerID="cri-o://9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe" gracePeriod=25 Oct 30 05:57:42.257271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.257213 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" podUID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" containerName="payload-pre-processing" containerID="cri-o://44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783" gracePeriod=25 Oct 30 05:57:42.404100 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.404038 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:57:42.409476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.409450 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:57:42.415082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.415036 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:57:42.493634 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493603 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.493817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493644 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.493817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493769 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.493817 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493797 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.494400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493829 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.494400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.494400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.493936 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.494729 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.494705 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:42.495340 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495053 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495565 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495658 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495690 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495736 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495763 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495784 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495810 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495840 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495913 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.496073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.495932 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496705 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496776 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496854 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496896 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496951 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.496985 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\") pod \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\" (UID: \"6c6f1a87-dc64-4453-9f4f-16e4b28c0436\") " Oct 30 05:57:42.497127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.497027 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.498199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.497589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\") pod \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\" (UID: \"987e8c69-d497-4b8f-b727-21ac43bb7bb4\") " Oct 30 05:57:42.498199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.497693 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.498199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.497778 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\") pod \"a1fd8c76-8881-4965-9942-b5615457dc75\" (UID: \"a1fd8c76-8881-4965-9942-b5615457dc75\") " Oct 30 05:57:42.499435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.499166 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.499435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.499193 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.500001 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.499727 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:42.502362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.501708 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.502362 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.502119 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:42.502774 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.502754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.503129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.503037 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.503418 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.503399 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.513722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.513235 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "kube-api-access-k6twq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.514883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.513869 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.515620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.515189 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.515915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.515778 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.515915 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.515802 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.516088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.516013 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:42.516306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.516274 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.516306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.516286 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "kube-api-access-gzrqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.516465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.516352 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.516465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.516364 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.517435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.517403 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:42.518642 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.518609 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.520901 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.520874 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.533352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.532107 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "kube-api-access-hdgq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.533352 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.532211 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token" pod "987e8c69-d497-4b8f-b727-21ac43bb7bb4" (UID: "987e8c69-d497-4b8f-b727-21ac43bb7bb4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.533713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.533675 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:57:42.533931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.533899 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:57:42.534202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.534170 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token" pod "a1fd8c76-8881-4965-9942-b5615457dc75" (UID: "a1fd8c76-8881-4965-9942-b5615457dc75"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.536902 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.536866 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token" pod "6c6f1a87-dc64-4453-9f4f-16e4b28c0436" (UID: "6c6f1a87-dc64-4453-9f4f-16e4b28c0436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.549906 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.549880 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:57:42.600260 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.599238 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601436 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601461 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601480 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601494 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzrqh\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-kube-api-access-gzrqh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601508 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601521 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601533 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601546 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601559 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601571 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601582 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601595 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1fd8c76-8881-4965-9942-b5615457dc75-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601610 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1fd8c76-8881-4965-9942-b5615457dc75-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601622 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601633 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601645 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601657 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601671 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6twq\" (UniqueName: \"kubernetes.io/projected/a1fd8c76-8881-4965-9942-b5615457dc75-kube-api-access-k6twq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601686 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601698 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601710 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601721 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1fd8c76-8881-4965-9942-b5615457dc75-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601732 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c6f1a87-dc64-4453-9f4f-16e4b28c0436-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601743 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdgq8\" (UniqueName: \"kubernetes.io/projected/987e8c69-d497-4b8f-b727-21ac43bb7bb4-kube-api-access-hdgq8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.602009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.601755 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/987e8c69-d497-4b8f-b727-21ac43bb7bb4-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.637870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.637836 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:57:42.672269 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.672235 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:57:42.703683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.703062 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\" (UniqueName: \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\") pod \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " Oct 30 05:57:42.703683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.703224 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\") pod \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\" (UID: \"1db2af24-5eb9-40fb-bdaa-41272710c8fe\") " Oct 30 05:57:42.704900 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.704869 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume" pod "1db2af24-5eb9-40fb-bdaa-41272710c8fe" (UID: "1db2af24-5eb9-40fb-bdaa-41272710c8fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:42.713213 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.713169 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2" pod "1db2af24-5eb9-40fb-bdaa-41272710c8fe" (UID: "1db2af24-5eb9-40fb-bdaa-41272710c8fe"). InnerVolumeSpecName "kube-api-access-vrjz2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.807130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.806094 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\" (UniqueName: \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\") pod \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " Oct 30 05:57:42.807130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.806238 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\") pod \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\" (UID: \"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4\") " Oct 30 05:57:42.807130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.807077 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrjz2\" (UniqueName: \"kubernetes.io/projected/1db2af24-5eb9-40fb-bdaa-41272710c8fe-kube-api-access-vrjz2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.807130 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.807101 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1db2af24-5eb9-40fb-bdaa-41272710c8fe-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.807481 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.807296 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume" pod "16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" (UID: "16edc878-2b95-48dc-8aa5-5dbe28a9a2c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:42.810940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.810897 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn" pod "16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" (UID: "16edc878-2b95-48dc-8aa5-5dbe28a9a2c4"). InnerVolumeSpecName "kube-api-access-gspcn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:42.928144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.921020 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspcn\" (UniqueName: \"kubernetes.io/projected/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-kube-api-access-gspcn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.928144 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.921077 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:42.966131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.960435 6171 generic.go:356] "Generic (PLEG): container finished" podID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" containerID="9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe" exitCode=0 Oct 30 05:57:42.966131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.960826 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" event={"ID":"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4","Type":"ContainerDied","Data":"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe"} Oct 30 05:57:42.966131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.960885 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" event={"ID":"16edc878-2b95-48dc-8aa5-5dbe28a9a2c4","Type":"ContainerDied","Data":"5844f1bdefaa52425e8f5d77f1d0afe3c67c6e6203e396f57a60f6a0fe0855a8"} Oct 30 05:57:42.966131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.960916 6171 scope.go:122] "RemoveContainer" containerID="9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe" Oct 30 05:57:42.966131 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.961112 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5" Oct 30 05:57:42.982120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.982039 6171 generic.go:356] "Generic (PLEG): container finished" podID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" containerID="44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783" exitCode=0 Oct 30 05:57:42.982412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.982392 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" Oct 30 05:57:42.988071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.982523 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" event={"ID":"1db2af24-5eb9-40fb-bdaa-41272710c8fe","Type":"ContainerDied","Data":"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783"} Oct 30 05:57:42.988071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:42.983764 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b" event={"ID":"1db2af24-5eb9-40fb-bdaa-41272710c8fe","Type":"ContainerDied","Data":"fab5a6b6f3866cb0f1e13c11e8077212f9b9579224c54a2d7e755dd525c8c09d"} Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.190588 6171 generic.go:356] "Generic (PLEG): container finished" podID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerID="b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea" exitCode=0 Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.190779 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" event={"ID":"6c6f1a87-dc64-4453-9f4f-16e4b28c0436","Type":"ContainerDied","Data":"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea"} Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.190833 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" event={"ID":"6c6f1a87-dc64-4453-9f4f-16e4b28c0436","Type":"ContainerDied","Data":"5f79363b823a39be58e7f8ad49bddca7137d40f7932a6796ab15e8efbc01b6f3"} Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.190917 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn" Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.208535 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:57:43.228263 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.223946 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d4ccc5-66cf5d675f-hl6r5"] Oct 30 05:57:43.265676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.240789 6171 generic.go:356] "Generic (PLEG): container finished" podID="a1fd8c76-8881-4965-9942-b5615457dc75" containerID="1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd" exitCode=0 Oct 30 05:57:43.265676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.240905 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" event={"ID":"a1fd8c76-8881-4965-9942-b5615457dc75","Type":"ContainerDied","Data":"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd"} Oct 30 05:57:43.265676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.240953 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" event={"ID":"a1fd8c76-8881-4965-9942-b5615457dc75","Type":"ContainerDied","Data":"f873409f1cfecc83d02b5cb9408e3bacba8c6c64b3af49552aec15455ea5011a"} Oct 30 05:57:43.265676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.241057 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54" Oct 30 05:57:43.281105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.272733 6171 generic.go:356] "Generic (PLEG): container finished" podID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerID="86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945" exitCode=0 Oct 30 05:57:43.281105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.272815 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" event={"ID":"987e8c69-d497-4b8f-b727-21ac43bb7bb4","Type":"ContainerDied","Data":"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945"} Oct 30 05:57:43.281105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.272858 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" event={"ID":"987e8c69-d497-4b8f-b727-21ac43bb7bb4","Type":"ContainerDied","Data":"4fa9880f6d40d2f83d29146b509698fac710ee0cff8ee75ed83355f90875baf1"} Oct 30 05:57:43.281105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.272939 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z" Oct 30 05:57:43.636159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.636098 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:57:43.754765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.734288 6171 scope.go:122] "RemoveContainer" containerID="9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe" Oct 30 05:57:43.754765 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:43.734814 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe\": container with ID starting with 9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe not found: ID does not exist" containerID="9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe" Oct 30 05:57:43.754765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.734869 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe"} err="failed to get container status \"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe\": rpc error: code = NotFound desc = could not find container \"9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe\": container with ID starting with 9e1b06f9c345eaea025e8d7716285b7449df9cff98dee6c4ea5122e1e41cd7fe not found: ID does not exist" Oct 30 05:57:43.754765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.734895 6171 scope.go:122] "RemoveContainer" containerID="44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783" Oct 30 05:57:43.982688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.982648 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" path="/var/lib/kubelet/pods/16edc878-2b95-48dc-8aa5-5dbe28a9a2c4/volumes" Oct 30 05:57:43.983596 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.983572 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-snphn"] Oct 30 05:57:43.983794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.983782 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:57:43.989289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.989267 6171 scope.go:122] "RemoveContainer" containerID="44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783" Oct 30 05:57:43.990014 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:43.989990 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783\": container with ID starting with 44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783 not found: ID does not exist" containerID="44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783" Oct 30 05:57:43.990436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.990389 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783"} err="failed to get container status \"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783\": rpc error: code = NotFound desc = could not find container \"44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783\": container with ID starting with 44da797e3d357c3dfcd3c86381c1efb8719159c2ff43df855cd6023449374783 not found: ID does not exist" Oct 30 05:57:43.990579 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.990563 6171 scope.go:122] "RemoveContainer" containerID="b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea" Oct 30 05:57:43.993435 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:43.993415 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d4ccc5-555c7f785f-2vv6b"] Oct 30 05:57:44.003095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.001986 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:57:44.012997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.011583 6171 scope.go:122] "RemoveContainer" containerID="b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea" Oct 30 05:57:44.012997 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:44.011993 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea\": container with ID starting with b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea not found: ID does not exist" containerID="b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea" Oct 30 05:57:44.012997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.012027 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea"} err="failed to get container status \"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea\": rpc error: code = NotFound desc = could not find container \"b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea\": container with ID starting with b46e3d96bb50012ed36e72a94d4933b52e6b117c2cdb36ec50b43f45b001f8ea not found: ID does not exist" Oct 30 05:57:44.012997 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.012657 6171 scope.go:122] "RemoveContainer" containerID="1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd" Oct 30 05:57:44.014578 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.013998 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-ffx54"] Oct 30 05:57:44.018124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.018090 6171 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","burstable","pod2afa0d1a-5836-4c90-8af6-5572f0979b9f"] err="unable to destroy cgroup paths for cgroup [kubepods burstable pod2afa0d1a-5836-4c90-8af6-5572f0979b9f] : Timed out while waiting for systemd to remove kubepods-burstable-pod2afa0d1a_5836_4c90_8af6_5572f0979b9f.slice" Oct 30 05:57:44.018242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.018125 6171 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="2afa0d1a-5836-4c90-8af6-5572f0979b9f" err="failed to delete cgroup paths for [kubepods burstable pod2afa0d1a-5836-4c90-8af6-5572f0979b9f] : unable to destroy cgroup paths for cgroup [kubepods burstable pod2afa0d1a-5836-4c90-8af6-5572f0979b9f] : Timed out while waiting for systemd to remove kubepods-burstable-pod2afa0d1a_5836_4c90_8af6_5572f0979b9f.slice" Oct 30 05:57:44.020736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.020708 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:57:44.026638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.026613 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-f6c7z"] Oct 30 05:57:44.028863 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.028843 6171 scope.go:122] "RemoveContainer" containerID="1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd" Oct 30 05:57:44.029222 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:44.029202 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd\": container with ID starting with 1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd not found: ID does not exist" containerID="1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd" Oct 30 05:57:44.029306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.029232 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd"} err="failed to get container status \"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd\": rpc error: code = NotFound desc = could not find container \"1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd\": container with ID starting with 1a6c3caeace50f77ccc63499aae8d52245c422ac604d170aa1486dee00d9e2fd not found: ID does not exist" Oct 30 05:57:44.029306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.029255 6171 scope.go:122] "RemoveContainer" containerID="86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945" Oct 30 05:57:44.045004 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.044960 6171 scope.go:122] "RemoveContainer" containerID="86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945" Oct 30 05:57:44.045429 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:44.045407 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945\": container with ID starting with 86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945 not found: ID does not exist" containerID="86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945" Oct 30 05:57:44.045569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:44.045434 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945"} err="failed to get container status \"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945\": rpc error: code = NotFound desc = could not find container \"86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945\": container with ID starting with 86993d42e8a231e4f488afa7b49b087f708f032f9f02808bb18b2a488f6c3945 not found: ID does not exist" Oct 30 05:57:45.676680 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:45.676644 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" path="/var/lib/kubelet/pods/1db2af24-5eb9-40fb-bdaa-41272710c8fe/volumes" Oct 30 05:57:45.677288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:45.677270 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" path="/var/lib/kubelet/pods/6c6f1a87-dc64-4453-9f4f-16e4b28c0436/volumes" Oct 30 05:57:45.677762 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:45.677738 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" path="/var/lib/kubelet/pods/987e8c69-d497-4b8f-b727-21ac43bb7bb4/volumes" Oct 30 05:57:45.678496 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:45.678477 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" path="/var/lib/kubelet/pods/a1fd8c76-8881-4965-9942-b5615457dc75/volumes" Oct 30 05:57:46.595398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:46.593714 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:57:46.595398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:46.593790 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:57:47.363469 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.363429 6171 generic.go:356] "Generic (PLEG): container finished" podID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerID="358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d" exitCode=0 Oct 30 05:57:47.364192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.363557 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerDied","Data":"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d"} Oct 30 05:57:47.365896 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.365876 6171 scope.go:122] "RemoveContainer" containerID="358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d" Oct 30 05:57:47.368364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.368318 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:47.368467 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.368421 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:47.368507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.368468 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:57:47.567147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.567092 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerName="payload-processing" containerID="cri-o://58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13" gracePeriod=25 Oct 30 05:57:47.641089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.640975 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" podUID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" containerName="payload-pre-processing" containerID="cri-o://095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712" gracePeriod=25 Oct 30 05:57:47.736086 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:47.736015 6171 logging.go:55] [core] [Channel #4102 SubChannel #4103]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.170:50052: connect: connection refused" Oct 30 05:57:47.992299 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.992275 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:57:47.996133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:47.996106 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:57:48.113727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.110781 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\" (UniqueName: \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\") pod \"43e73e7c-e59e-4af0-8678-4d9b82816849\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " Oct 30 05:57:48.113727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.110998 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\" (UniqueName: \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\") pod \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " Oct 30 05:57:48.113727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.111289 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\") pod \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\" (UID: \"26f0d9e5-89a3-4c65-ae67-d0989a78713e\") " Oct 30 05:57:48.113727 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.111986 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\") pod \"43e73e7c-e59e-4af0-8678-4d9b82816849\" (UID: \"43e73e7c-e59e-4af0-8678-4d9b82816849\") " Oct 30 05:57:48.119072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.117514 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume" pod "26f0d9e5-89a3-4c65-ae67-d0989a78713e" (UID: "26f0d9e5-89a3-4c65-ae67-d0989a78713e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:48.119072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.118329 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume" pod "43e73e7c-e59e-4af0-8678-4d9b82816849" (UID: "43e73e7c-e59e-4af0-8678-4d9b82816849"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:48.149413 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.149371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh" pod "43e73e7c-e59e-4af0-8678-4d9b82816849" (UID: "43e73e7c-e59e-4af0-8678-4d9b82816849"). InnerVolumeSpecName "kube-api-access-r27xh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:48.149736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.149715 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c" pod "26f0d9e5-89a3-4c65-ae67-d0989a78713e" (UID: "26f0d9e5-89a3-4c65-ae67-d0989a78713e"). InnerVolumeSpecName "kube-api-access-ql94c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:48.218400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.218308 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26f0d9e5-89a3-4c65-ae67-d0989a78713e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:48.218400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.218345 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43e73e7c-e59e-4af0-8678-4d9b82816849-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:48.218400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.218354 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r27xh\" (UniqueName: \"kubernetes.io/projected/43e73e7c-e59e-4af0-8678-4d9b82816849-kube-api-access-r27xh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:48.218400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.218363 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql94c\" (UniqueName: \"kubernetes.io/projected/26f0d9e5-89a3-4c65-ae67-d0989a78713e-kube-api-access-ql94c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:57:48.374925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.374890 6171 generic.go:356] "Generic (PLEG): container finished" podID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerID="58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13" exitCode=0 Oct 30 05:57:48.375412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.374949 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" Oct 30 05:57:48.375412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.374967 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" event={"ID":"43e73e7c-e59e-4af0-8678-4d9b82816849","Type":"ContainerDied","Data":"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13"} Oct 30 05:57:48.375412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.375058 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" event={"ID":"43e73e7c-e59e-4af0-8678-4d9b82816849","Type":"ContainerDied","Data":"b3a198e61da4aa7c3625f51f5eb27572b03d6e2a955764ce53d0680972e99561"} Oct 30 05:57:48.375412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.375090 6171 scope.go:122] "RemoveContainer" containerID="58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13" Oct 30 05:57:48.377717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.377695 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerStarted","Data":"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c"} Oct 30 05:57:48.379369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.379348 6171 generic.go:356] "Generic (PLEG): container finished" podID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" containerID="095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712" exitCode=0 Oct 30 05:57:48.379494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.379413 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" Oct 30 05:57:48.379494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.379479 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" event={"ID":"26f0d9e5-89a3-4c65-ae67-d0989a78713e","Type":"ContainerDied","Data":"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712"} Oct 30 05:57:48.379609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.379528 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk" event={"ID":"26f0d9e5-89a3-4c65-ae67-d0989a78713e","Type":"ContainerDied","Data":"8316d56ad3e2f7a3353752cdb2f89a5b58cd45f379875b6131e1b3da3d71e508"} Oct 30 05:57:48.416581 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.416556 6171 scope.go:122] "RemoveContainer" containerID="58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13" Oct 30 05:57:48.416934 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:48.416914 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13\": container with ID starting with 58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13 not found: ID does not exist" containerID="58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13" Oct 30 05:57:48.416989 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.416945 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13"} err="failed to get container status \"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13\": rpc error: code = NotFound desc = could not find container \"58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13\": container with ID starting with 58b0e0a3b1f5f356a55ec1d94d24c73e5c3f7d2d77a1576601ae27b1c8843a13 not found: ID does not exist" Oct 30 05:57:48.416989 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.416963 6171 scope.go:122] "RemoveContainer" containerID="095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712" Oct 30 05:57:48.428774 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.428756 6171 scope.go:122] "RemoveContainer" containerID="095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712" Oct 30 05:57:48.429055 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:48.429022 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712\": container with ID starting with 095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712 not found: ID does not exist" containerID="095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712" Oct 30 05:57:48.429128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.429063 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712"} err="failed to get container status \"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712\": rpc error: code = NotFound desc = could not find container \"095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712\": container with ID starting with 095a08ef7bd3c153d60cc135304fddb12ec83fd2f996829bb406be95dce96712 not found: ID does not exist" Oct 30 05:57:48.550267 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.550226 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:57:48.555551 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.555525 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc"] Oct 30 05:57:48.615193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.615165 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:57:48.622538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.622515 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-3571f8-c7855d6bc-4dkqk"] Oct 30 05:57:48.735406 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.735364 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-a-3571f8-765764b5dc-rxgxc" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.170:50052\" within 1s: context deadline exceeded" Oct 30 05:57:48.937507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.937403 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:57:48.937696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.937556 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:57:48.937696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:48.937622 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 05:57:49.674476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:49.674444 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" path="/var/lib/kubelet/pods/26f0d9e5-89a3-4c65-ae67-d0989a78713e/volumes" Oct 30 05:57:49.674880 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:49.674871 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" path="/var/lib/kubelet/pods/43e73e7c-e59e-4af0-8678-4d9b82816849/volumes" Oct 30 05:57:49.937451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:49.937364 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:57:49.937451 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:49.937436 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 05:57:50.543248 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.543210 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:57:50.544151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544130 6171 cpu_manager.go:414] "containerMap: removing container" podUID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" containerName="payload-processing" Oct 30 05:57:50.544151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544151 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" containerName="payload-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544174 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" containerName="revoke-keys" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544181 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" containerName="revoke-keys" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544199 6171 cpu_manager.go:414] "containerMap: removing container" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544206 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544222 6171 cpu_manager.go:414] "containerMap: removing container" podUID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" containerName="payload-pre-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544229 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" containerName="payload-pre-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544261 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544269 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544279 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" containerName="payload-pre-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544286 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" containerName="payload-pre-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544336 6171 cpu_manager.go:414] "containerMap: removing container" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerName="payload-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544344 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerName="payload-processing" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544358 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" Oct 30 05:57:50.544364 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544365 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544605 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="16edc878-2b95-48dc-8aa5-5dbe28a9a2c4" containerName="payload-processing" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544620 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" containerName="revoke-keys" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544633 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="987e8c69-d497-4b8f-b727-21ac43bb7bb4" containerName="istio-proxy" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544644 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="26f0d9e5-89a3-4c65-ae67-d0989a78713e" containerName="payload-pre-processing" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544658 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c6f1a87-dc64-4453-9f4f-16e4b28c0436" containerName="istio-proxy" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544667 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1db2af24-5eb9-40fb-bdaa-41272710c8fe" containerName="payload-pre-processing" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544687 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="43e73e7c-e59e-4af0-8678-4d9b82816849" containerName="payload-processing" Oct 30 05:57:50.545105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.544697 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1fd8c76-8881-4965-9942-b5615457dc75" containerName="istio-proxy" Oct 30 05:57:50.550249 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.550230 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.571717 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.571693 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1b2243ef-serving-cert\"" Oct 30 05:57:50.575472 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.575428 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:57:50.658261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.658235 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.658441 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.658299 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26kl\" (UniqueName: \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.658484 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.658460 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.658523 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.658500 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.759783 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.759752 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26kl\" (UniqueName: \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.760303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.759901 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.760303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.759956 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.760303 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.760210 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.760708 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.760688 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.763038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.763012 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.763038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.763034 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.820055 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.818122 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26kl\" (UniqueName: \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\") pod \"maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.877070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.875416 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:50.949075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.940646 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:57:50.949075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:50.940727 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 05:57:51.441719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:51.441688 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:57:51.443906 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:51.443879 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" event={"ID":"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2","Type":"ContainerStarted","Data":"46ba5f0b7f5e83db4d60f3761815c6b9496e6ccbfbec14bd512dc46b351f02a3"} Oct 30 05:57:51.953121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:51.944693 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:57:51.953121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:51.944804 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 05:57:52.474434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:52.474302 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" event={"ID":"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2","Type":"ContainerStarted","Data":"e97bd902ba00fb7b15727eef501d8cc8cf96f8985e2eb98019e31358b7706dae"} Oct 30 05:57:52.474739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:52.474722 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:52.534221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:52.534167 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" podStartSLOduration=2.071167248 podStartE2EDuration="2.534153273s" totalImagesPullingTime="462.986025ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:52.533282185 +0000 UTC m=+1530.973659376" watchObservedRunningTime="2026-10-30 05:57:52.534153273 +0000 UTC m=+1530.974530464" Oct 30 05:57:52.937242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:52.937159 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 05:57:52.937395 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:52.937233 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 05:57:53.951755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:53.951727 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:57:53.954231 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:53.954209 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:57:53.962720 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:53.957307 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 05:57:56.384179 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:57:56.384140 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:57:58.508398 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.508364 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:57:58.682425 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.682397 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 05:57:58.716276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.716237 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 05:57:58.716446 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.716330 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.735922 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.735890 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-de7242-serving-cert\"" Oct 30 05:57:58.806722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.806647 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.806722 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.806690 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.806919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.806779 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.806919 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.806827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf97l\" (UniqueName: \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.910382 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.910453 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf97l\" (UniqueName: \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.910882 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.910944 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.911843 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.926694 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.919309 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.932621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.932592 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:58.948092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:58.948014 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf97l\" (UniqueName: \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\") pod \"maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:59.049069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:59.043749 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:57:59.382604 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:57:59.382566 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce723f5_1758_4ccf_bd0b_426238b0d167.slice/crio-990ec9b6e161dcb676791c652e03c7ed108b0ed841eb404f58ebf370a20e43f8 WatchSource:0}: Error finding container 990ec9b6e161dcb676791c652e03c7ed108b0ed841eb404f58ebf370a20e43f8: Status 404 returned error can't find the container with id 990ec9b6e161dcb676791c652e03c7ed108b0ed841eb404f58ebf370a20e43f8 Oct 30 05:57:59.384329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:59.384304 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 05:57:59.645625 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:57:59.645552 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" event={"ID":"6ce723f5-1758-4ccf-bd0b-426238b0d167","Type":"ContainerStarted","Data":"990ec9b6e161dcb676791c652e03c7ed108b0ed841eb404f58ebf370a20e43f8"} Oct 30 05:58:00.733600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:00.733552 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" event={"ID":"6ce723f5-1758-4ccf-bd0b-426238b0d167","Type":"ContainerStarted","Data":"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f"} Oct 30 05:58:00.734165 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:00.733639 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:58:00.782142 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:00.782087 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" podStartSLOduration=2.3374168 podStartE2EDuration="2.78202641s" totalImagesPullingTime="444.60961ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:00.78024105 +0000 UTC m=+1539.220618242" watchObservedRunningTime="2026-10-30 05:58:00.78202641 +0000 UTC m=+1539.222403599" Oct 30 05:58:06.758185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:06.758154 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 05:58:08.034610 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.034576 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:58:08.035121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.034833 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" containerID="cri-o://b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd" gracePeriod=30 Oct 30 05:58:08.048112 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.048083 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:58:08.048338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.048300 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" containerID="cri-o://e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808" gracePeriod=30 Oct 30 05:58:08.053084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.053059 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:58:08.053246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.053228 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" containerID="cri-o://e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8" gracePeriod=30 Oct 30 05:58:08.061761 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.061736 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:58:08.061939 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:08.061920 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" containerID="cri-o://4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b" gracePeriod=30 Oct 30 05:58:12.158070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.157110 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:58:12.158070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.157176 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:58:12.322236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.322201 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-rxgds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:58:12.322509 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.322478 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-rxgds" podUID="b78e85c7-fea8-49e0-af5a-802c1ebe3ea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.42:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:58:12.896276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.896240 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-2c9x4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:58:12.896600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:12.896559 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:58:13.440871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.440520 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:58:13.485367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.485237 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:58:13.524139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524096 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.524330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524150 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.524330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524256 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.524463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524347 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.524616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524591 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.524755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.524687 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.525039 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:13.525127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525114 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.525460 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525437 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.525541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525505 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.525595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525555 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.525595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.525587 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\") pod \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\" (UID: \"048f2d6f-f420-4e1e-8a9e-b922c5b785dc\") " Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.526736 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.526793 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.527035 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.527079 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.527095 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.527110 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.527145 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.527123 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.528644 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.528624 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.528894 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.528871 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.529582 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.529553 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "kube-api-access-dtgw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.529976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.529951 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo" pod "048f2d6f-f420-4e1e-8a9e-b922c5b785dc" (UID: "048f2d6f-f420-4e1e-8a9e-b922c5b785dc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.557883 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.557812 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:58:13.561327 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.561305 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628162 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628219 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628265 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628332 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628431 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628464 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628519 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628590 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.629453 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.628714 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\") pod \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\" (UID: \"1f605e3c-ee03-49ba-b55d-f999c2c6684b\") " Oct 30 05:58:13.630089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.629953 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.630089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.629977 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.630089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.629990 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtgw5\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-kube-api-access-dtgw5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.630089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.630004 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/048f2d6f-f420-4e1e-8a9e-b922c5b785dc-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.630367 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.630342 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.630685 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.630658 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:13.631304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.631269 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.631693 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.631671 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.631869 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.631828 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.650454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.650409 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.650454 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.650438 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.650658 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.650509 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.652698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.652674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h" pod "1f605e3c-ee03-49ba-b55d-f999c2c6684b" (UID: "1f605e3c-ee03-49ba-b55d-f999c2c6684b"). InnerVolumeSpecName "kube-api-access-xb44h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730713 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730795 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730835 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730867 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730934 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.730962 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731102 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731186 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731225 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731327 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731439 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731484 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731524 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731580 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731659 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\") pod \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\" (UID: \"c2a59f97-740c-42a0-a9fa-29e7eb665bd7\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731693 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731731 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.731777 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\") pod \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\" (UID: \"99c9d95e-7e51-4569-bd0b-e01bf5378d53\") " Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732557 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732582 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732597 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732610 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb44h\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-kube-api-access-xb44h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732624 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732637 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732651 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732665 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1f605e3c-ee03-49ba-b55d-f999c2c6684b-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732678 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1f605e3c-ee03-49ba-b55d-f999c2c6684b-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.732910 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.735361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.735528 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.735690 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.735703 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.736327 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.737142 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.737424 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.737957 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.737997 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "kube-api-access-mp2wl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.738260 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.739407 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.738544 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:13.741807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.739970 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.741807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.741453 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.742412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.742387 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy" pod "c2a59f97-740c-42a0-a9fa-29e7eb665bd7" (UID: "c2a59f97-740c-42a0-a9fa-29e7eb665bd7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:13.742505 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.742416 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "kube-api-access-v4hpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.757360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.757333 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:13.770337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.770318 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token" pod "99c9d95e-7e51-4569-bd0b-e01bf5378d53" (UID: "99c9d95e-7e51-4569-bd0b-e01bf5378d53"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:13.834921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834875 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4hpw\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-kube-api-access-v4hpw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.834921 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834921 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834937 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834952 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834965 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834979 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mp2wl\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-kube-api-access-mp2wl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.834995 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835012 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835025 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835039 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835096 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835109 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835117 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835125 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835134 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835142 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835150 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99c9d95e-7e51-4569-bd0b-e01bf5378d53-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:13.835177 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:13.835157 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2a59f97-740c-42a0-a9fa-29e7eb665bd7-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.145374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.145329 6171 generic.go:356] "Generic (PLEG): container finished" podID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerID="e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808" exitCode=0 Oct 30 05:58:14.145633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.145432 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" event={"ID":"1f605e3c-ee03-49ba-b55d-f999c2c6684b","Type":"ContainerDied","Data":"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808"} Oct 30 05:58:14.145633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.145482 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" event={"ID":"1f605e3c-ee03-49ba-b55d-f999c2c6684b","Type":"ContainerDied","Data":"2bd140deb3f01700bbed4fe31ccfdc66079c332ac4033e6096f15738e456d09f"} Oct 30 05:58:14.145633 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.145511 6171 scope.go:122] "RemoveContainer" containerID="e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808" Oct 30 05:58:14.145930 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.145677 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5" Oct 30 05:58:14.149012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.148981 6171 generic.go:356] "Generic (PLEG): container finished" podID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerID="e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8" exitCode=0 Oct 30 05:58:14.149247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.149082 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" event={"ID":"99c9d95e-7e51-4569-bd0b-e01bf5378d53","Type":"ContainerDied","Data":"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8"} Oct 30 05:58:14.149247 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.149141 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" event={"ID":"99c9d95e-7e51-4569-bd0b-e01bf5378d53","Type":"ContainerDied","Data":"b266dd782f8d930a5cee1489b6b124c6093c814b327ec92ac526b24a299c11ed"} Oct 30 05:58:14.149463 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.149441 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl" Oct 30 05:58:14.150671 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.150649 6171 generic.go:356] "Generic (PLEG): container finished" podID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerID="b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd" exitCode=0 Oct 30 05:58:14.150786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.150712 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" Oct 30 05:58:14.150786 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.150746 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" event={"ID":"048f2d6f-f420-4e1e-8a9e-b922c5b785dc","Type":"ContainerDied","Data":"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd"} Oct 30 05:58:14.150898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.150792 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl" event={"ID":"048f2d6f-f420-4e1e-8a9e-b922c5b785dc","Type":"ContainerDied","Data":"a2064406831e3f4b90d8b7b6456908a324e7634e5fff298a93263dc1baabd0a0"} Oct 30 05:58:14.152288 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.152268 6171 generic.go:356] "Generic (PLEG): container finished" podID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerID="4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b" exitCode=0 Oct 30 05:58:14.152438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.152310 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" Oct 30 05:58:14.152438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.152322 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" event={"ID":"c2a59f97-740c-42a0-a9fa-29e7eb665bd7","Type":"ContainerDied","Data":"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b"} Oct 30 05:58:14.152438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.152363 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp" event={"ID":"c2a59f97-740c-42a0-a9fa-29e7eb665bd7","Type":"ContainerDied","Data":"2739eebf36aff604ac2846a8c2a90dfc27553df5f64ab4fd4641d91c0e7e2bcd"} Oct 30 05:58:14.164159 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.163969 6171 scope.go:122] "RemoveContainer" containerID="e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808" Oct 30 05:58:14.164324 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:14.164299 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808\": container with ID starting with e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808 not found: ID does not exist" containerID="e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808" Oct 30 05:58:14.164409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.164334 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808"} err="failed to get container status \"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808\": rpc error: code = NotFound desc = could not find container \"e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808\": container with ID starting with e04c8c94a31d2d50bde0ebf2be0856017a7f50a30a33de9f3437bd29c7b06808 not found: ID does not exist" Oct 30 05:58:14.164409 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.164354 6171 scope.go:122] "RemoveContainer" containerID="e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8" Oct 30 05:58:14.175736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.175720 6171 scope.go:122] "RemoveContainer" containerID="e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8" Oct 30 05:58:14.176066 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:14.176012 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8\": container with ID starting with e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8 not found: ID does not exist" containerID="e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8" Oct 30 05:58:14.176169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.176065 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8"} err="failed to get container status \"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8\": rpc error: code = NotFound desc = could not find container \"e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8\": container with ID starting with e730be3f1b93112f388427deaabba2b4a33fff13e43b902f93581a607506c4e8 not found: ID does not exist" Oct 30 05:58:14.176169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.176089 6171 scope.go:122] "RemoveContainer" containerID="b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd" Oct 30 05:58:14.187603 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.187588 6171 scope.go:122] "RemoveContainer" containerID="b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd" Oct 30 05:58:14.187841 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:14.187821 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd\": container with ID starting with b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd not found: ID does not exist" containerID="b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd" Oct 30 05:58:14.187912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.187846 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd"} err="failed to get container status \"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd\": rpc error: code = NotFound desc = could not find container \"b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd\": container with ID starting with b651f6b73958b79290050be4c3e028909ca9e65004e910ba1658e758422619cd not found: ID does not exist" Oct 30 05:58:14.187912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.187861 6171 scope.go:122] "RemoveContainer" containerID="4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b" Oct 30 05:58:14.229298 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.229253 6171 scope.go:122] "RemoveContainer" containerID="4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b" Oct 30 05:58:14.229836 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:14.229696 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b\": container with ID starting with 4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b not found: ID does not exist" containerID="4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b" Oct 30 05:58:14.229836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.229735 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b"} err="failed to get container status \"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b\": rpc error: code = NotFound desc = could not find container \"4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b\": container with ID starting with 4dc9a86f45056efb04d8a4e4dec126fb8d8098349d700c27c096884a5b3a0d0b not found: ID does not exist" Oct 30 05:58:14.286076 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.279355 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:58:14.298152 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.298106 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-525h5"] Oct 30 05:58:14.307261 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.307223 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:58:14.316543 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.316481 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-rrhgl"] Oct 30 05:58:14.322556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.322516 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:58:14.328239 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.328191 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-qdgvp"] Oct 30 05:58:14.375953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.375889 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:58:14.404274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:14.404249 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-67pwl"] Oct 30 05:58:15.675427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:15.675395 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" path="/var/lib/kubelet/pods/048f2d6f-f420-4e1e-8a9e-b922c5b785dc/volumes" Oct 30 05:58:15.675931 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:15.675908 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" path="/var/lib/kubelet/pods/1f605e3c-ee03-49ba-b55d-f999c2c6684b/volumes" Oct 30 05:58:15.676436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:15.676421 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" path="/var/lib/kubelet/pods/99c9d95e-7e51-4569-bd0b-e01bf5378d53/volumes" Oct 30 05:58:15.677161 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:15.677146 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" path="/var/lib/kubelet/pods/c2a59f97-740c-42a0-a9fa-29e7eb665bd7/volumes" Oct 30 05:58:25.591137 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.589255 6171 scope.go:122] "RemoveContainer" containerID="efc32d916f532891f80f2780aeb44a84a4023a2de7503543141a7e1458ea107c" Oct 30 05:58:25.615705 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.615682 6171 scope.go:122] "RemoveContainer" containerID="2259620fc6c0bfdb76f76ef1f964d09b50b620511a1237c863af656b8b0df419" Oct 30 05:58:25.631951 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.631929 6171 scope.go:122] "RemoveContainer" containerID="2d7a2c751ea46cecde482ef1a2d1f84320b67561cc22f46ee16d7a6b077910ca" Oct 30 05:58:25.642279 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.642261 6171 scope.go:122] "RemoveContainer" containerID="776f33a5812825473400cdab9ce728154731987df55934b6ba9a8a93f91592db" Oct 30 05:58:25.655565 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.655543 6171 scope.go:122] "RemoveContainer" containerID="2e2ca50881b77f08ecdcf7d398689c6e741503f81ccf8392c68e3f04acbba538" Oct 30 05:58:25.670148 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.668232 6171 scope.go:122] "RemoveContainer" containerID="c73a2560163edf94458004fbc62d15593b7eb81b5e967fa7b402eb0373b2e048" Oct 30 05:58:25.694125 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.694108 6171 scope.go:122] "RemoveContainer" containerID="b0f17496730c18917520c0e4268a3439eb3f80e03c2adacbf2b87b34b7d8e28d" Oct 30 05:58:25.703881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.703850 6171 scope.go:122] "RemoveContainer" containerID="b19fb05cb4cace52c3656e6b23614ba05f2063f96ab8a1b832453a51a18f0a35" Oct 30 05:58:25.713245 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:25.713227 6171 scope.go:122] "RemoveContainer" containerID="c662a04bb750cebb0467c7d4f71dafbb752e84a95b81c0576d4f637ba6338836" Oct 30 05:58:26.381981 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:26.381943 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:58:30.776944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.776914 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-9201-d4e88d"] Oct 30 05:58:30.777986 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.777961 6171 cpu_manager.go:414] "containerMap: removing container" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" Oct 30 05:58:30.777986 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.777985 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778031 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778038 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778087 6171 cpu_manager.go:414] "containerMap: removing container" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778094 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778130 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" Oct 30 05:58:30.778188 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778137 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" Oct 30 05:58:30.778471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778337 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2a59f97-740c-42a0-a9fa-29e7eb665bd7" containerName="istio-proxy" Oct 30 05:58:30.778471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778358 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c9d95e-7e51-4569-bd0b-e01bf5378d53" containerName="istio-proxy" Oct 30 05:58:30.778471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778374 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f605e3c-ee03-49ba-b55d-f999c2c6684b" containerName="istio-proxy" Oct 30 05:58:30.778471 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.778389 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="048f2d6f-f420-4e1e-8a9e-b922c5b785dc" containerName="istio-proxy" Oct 30 05:58:30.785935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.785916 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:30.800215 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.800193 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9201-d4e88d"] Oct 30 05:58:30.808444 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.808421 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-4m56v\"" Oct 30 05:58:30.871119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.871098 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mqjn\" (UniqueName: \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\") pod \"test-curl-9201-d4e88d\" (UID: \"8d94072e-e718-4159-8d65-ec7f88b3b696\") " pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:30.972842 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.972819 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mqjn\" (UniqueName: \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\") pod \"test-curl-9201-d4e88d\" (UID: \"8d94072e-e718-4159-8d65-ec7f88b3b696\") " pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:30.998532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:30.998509 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mqjn\" (UniqueName: \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\") pod \"test-curl-9201-d4e88d\" (UID: \"8d94072e-e718-4159-8d65-ec7f88b3b696\") " pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:31.102292 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:31.102237 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:31.412948 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:58:31.412917 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d94072e_e718_4159_8d65_ec7f88b3b696.slice/crio-b55073c992d65f285e7271a91250931f4ef87f7710802bdd195c82ff2251f0d6 WatchSource:0}: Error finding container b55073c992d65f285e7271a91250931f4ef87f7710802bdd195c82ff2251f0d6: Status 404 returned error can't find the container with id b55073c992d65f285e7271a91250931f4ef87f7710802bdd195c82ff2251f0d6 Oct 30 05:58:31.413514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:31.413487 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-9201-d4e88d"] Oct 30 05:58:32.407773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:32.407743 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9201-d4e88d" event={"ID":"8d94072e-e718-4159-8d65-ec7f88b3b696","Type":"ContainerStarted","Data":"897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f"} Oct 30 05:58:32.408170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:32.407782 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-9201-d4e88d" event={"ID":"8d94072e-e718-4159-8d65-ec7f88b3b696","Type":"ContainerStarted","Data":"b55073c992d65f285e7271a91250931f4ef87f7710802bdd195c82ff2251f0d6"} Oct 30 05:58:32.451171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:32.451116 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-9201-d4e88d" podStartSLOduration=2.451099625 podStartE2EDuration="2.451099625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:32.449446296 +0000 UTC m=+1570.889823487" watchObservedRunningTime="2026-10-30 05:58:32.451099625 +0000 UTC m=+1570.891476817" Oct 30 05:58:33.347392 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:33.347358 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-9201-d4e88d"] Oct 30 05:58:33.351996 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:33.351970 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-9201-d4e88d"] Oct 30 05:58:34.425083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:34.425014 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-9201-d4e88d" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" containerName="test-curl-9201-d4e88d" containerID="cri-o://897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f" gracePeriod=2 Oct 30 05:58:36.663234 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:36.663212 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:36.689083 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:36.688981 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\" (UniqueName: \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\") pod \"8d94072e-e718-4159-8d65-ec7f88b3b696\" (UID: \"8d94072e-e718-4159-8d65-ec7f88b3b696\") " Oct 30 05:58:36.691095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:36.691036 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn" pod "8d94072e-e718-4159-8d65-ec7f88b3b696" (UID: "8d94072e-e718-4159-8d65-ec7f88b3b696"). InnerVolumeSpecName "kube-api-access-9mqjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.790871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:36.790847 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mqjn\" (UniqueName: \"kubernetes.io/projected/8d94072e-e718-4159-8d65-ec7f88b3b696-kube-api-access-9mqjn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:37.045120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.045015 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl"] Oct 30 05:58:37.058954 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.058927 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d8a893-xptcl"] Oct 30 05:58:37.453196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.453116 6171 generic.go:356] "Generic (PLEG): container finished" podID="8d94072e-e718-4159-8d65-ec7f88b3b696" containerID="897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f" exitCode=137 Oct 30 05:58:37.453196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.453159 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:37.453196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.453194 6171 scope.go:122] "RemoveContainer" containerID="897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f" Oct 30 05:58:37.465427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.465410 6171 scope.go:122] "RemoveContainer" containerID="897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f" Oct 30 05:58:37.465714 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:37.465693 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f\": container with ID starting with 897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f not found: ID does not exist" containerID="897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f" Oct 30 05:58:37.465788 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.465722 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f"} err="failed to get container status \"897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f\": rpc error: code = NotFound desc = could not find container \"897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f\": container with ID starting with 897d4daa19516e9c1b93c34d581c43e4263a4b359ee45ef0a61cfcb57e3dcf6f not found: ID does not exist" Oct 30 05:58:37.477427 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:37.477400 6171 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-9201-d4e88d\" is forbidden: User \"system:node:ip-10-0-136-35.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-35.ec2.internal' and this object" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" pod="opendatahub/test-curl-9201-d4e88d" Oct 30 05:58:37.690739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.690708 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc29184-99af-4928-8b56-472bbaa7c1f9" path="/var/lib/kubelet/pods/7fc29184-99af-4928-8b56-472bbaa7c1f9/volumes" Oct 30 05:58:37.691264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:37.691248 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" path="/var/lib/kubelet/pods/8d94072e-e718-4159-8d65-ec7f88b3b696/volumes" Oct 30 05:58:39.278537 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.278504 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 05:58:39.279925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.279900 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" containerName="test-curl-9201-d4e88d" Oct 30 05:58:39.279925 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.279927 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" containerName="test-curl-9201-d4e88d" Oct 30 05:58:39.280173 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.280159 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d94072e-e718-4159-8d65-ec7f88b3b696" containerName="test-curl-9201-d4e88d" Oct 30 05:58:39.285914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.285888 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.303976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.303949 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 05:58:39.304408 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.304386 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-862dc2\"/\"openshift-service-ca.crt\"" Oct 30 05:58:39.311343 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.311321 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-862dc2\"/\"kube-root-ca.crt\"" Oct 30 05:58:39.311465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.311362 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-862dc2\"/\"test-model-862dc2-kserve-self-signed-certs\"" Oct 30 05:58:39.311556 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.311515 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-862dc2\"/\"default-dockercfg-8f9gg\"" Oct 30 05:58:39.335706 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.335682 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.335795 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.335746 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.335864 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.335848 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.335905 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.335881 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.336025 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.336010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.336090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.336076 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzk4n\" (UniqueName: \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.437966 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.437933 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438035 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438132 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzk4n\" (UniqueName: \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438292 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438328 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438419 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438467 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.438689 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438636 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.439009 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.438837 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.440663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.440642 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.440756 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.440729 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.463335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.463314 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzk4n\" (UniqueName: \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\") pod \"test-model-862dc2-kserve-dd7448f-qbkjx\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:39.615118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:39.610353 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:40.064561 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:58:40.064526 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5f8469a_a47d_4167_9c92_e157b60f4508.slice/crio-3f06cd664fda62dc9454bf3696ef94b20d431eddd63d48fff886accf896bc810 WatchSource:0}: Error finding container 3f06cd664fda62dc9454bf3696ef94b20d431eddd63d48fff886accf896bc810: Status 404 returned error can't find the container with id 3f06cd664fda62dc9454bf3696ef94b20d431eddd63d48fff886accf896bc810 Oct 30 05:58:40.065399 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:40.065376 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 05:58:40.484222 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:40.484187 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" event={"ID":"a5f8469a-a47d-4167-9c92-e157b60f4508","Type":"ContainerStarted","Data":"fffde35bd487a21e9cc787707def8e80f23cedfe5364891da52a563f3319e57f"} Oct 30 05:58:40.484619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:40.484231 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" event={"ID":"a5f8469a-a47d-4167-9c92-e157b60f4508","Type":"ContainerStarted","Data":"3f06cd664fda62dc9454bf3696ef94b20d431eddd63d48fff886accf896bc810"} Oct 30 05:58:40.535590 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:40.535532 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podStartSLOduration=1.5355180210000001 podStartE2EDuration="1.535518021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:40.530403015 +0000 UTC m=+1578.970780232" watchObservedRunningTime="2026-10-30 05:58:40.535518021 +0000 UTC m=+1578.975895213" Oct 30 05:58:40.611424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:40.611397 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:42.709287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:42.709250 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:58:42.709698 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:42.709492 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" podUID="13896cfb-594f-4652-ac3e-6dc1940da08b" containerName="maas-api" containerID="cri-o://802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de" gracePeriod=30 Oct 30 05:58:43.333084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.331457 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:58:43.333084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.332635 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\") pod \"13896cfb-594f-4652-ac3e-6dc1940da08b\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " Oct 30 05:58:43.333084 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.332789 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\") pod \"13896cfb-594f-4652-ac3e-6dc1940da08b\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " Oct 30 05:58:43.333465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.333148 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\" (UniqueName: \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\") pod \"13896cfb-594f-4652-ac3e-6dc1940da08b\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " Oct 30 05:58:43.333465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.333392 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\") pod \"13896cfb-594f-4652-ac3e-6dc1940da08b\" (UID: \"13896cfb-594f-4652-ac3e-6dc1940da08b\") " Oct 30 05:58:43.356940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.351597 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca" pod "13896cfb-594f-4652-ac3e-6dc1940da08b" (UID: "13896cfb-594f-4652-ac3e-6dc1940da08b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:43.356940 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.354677 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13896cfb-594f-4652-ac3e-6dc1940da08b-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:43.372073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.361550 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc" pod "13896cfb-594f-4652-ac3e-6dc1940da08b" (UID: "13896cfb-594f-4652-ac3e-6dc1940da08b"). InnerVolumeSpecName "kube-api-access-5zptc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:43.397071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.394215 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls" pod "13896cfb-594f-4652-ac3e-6dc1940da08b" (UID: "13896cfb-594f-4652-ac3e-6dc1940da08b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:43.397071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.394412 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs" pod "13896cfb-594f-4652-ac3e-6dc1940da08b" (UID: "13896cfb-594f-4652-ac3e-6dc1940da08b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:43.462218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.459606 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:43.462218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.459643 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13896cfb-594f-4652-ac3e-6dc1940da08b-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:43.462218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.459658 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zptc\" (UniqueName: \"kubernetes.io/projected/13896cfb-594f-4652-ac3e-6dc1940da08b-kube-api-access-5zptc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:43.577424 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.577378 6171 generic.go:356] "Generic (PLEG): container finished" podID="13896cfb-594f-4652-ac3e-6dc1940da08b" containerID="802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de" exitCode=0 Oct 30 05:58:43.577614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.577434 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" Oct 30 05:58:43.577614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.577464 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" event={"ID":"13896cfb-594f-4652-ac3e-6dc1940da08b","Type":"ContainerDied","Data":"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de"} Oct 30 05:58:43.577614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.577526 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf" event={"ID":"13896cfb-594f-4652-ac3e-6dc1940da08b","Type":"ContainerDied","Data":"0fc1d435e6e9590f4861fed658e6991c4dcfe830d57750250ef8e4a51dea63f5"} Oct 30 05:58:43.577614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.577558 6171 scope.go:122] "RemoveContainer" containerID="802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de" Oct 30 05:58:43.591420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.591394 6171 scope.go:122] "RemoveContainer" containerID="802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de" Oct 30 05:58:43.591753 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:43.591730 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de\": container with ID starting with 802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de not found: ID does not exist" containerID="802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de" Oct 30 05:58:43.591844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.591768 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de"} err="failed to get container status \"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de\": rpc error: code = NotFound desc = could not find container \"802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de\": container with ID starting with 802bef6d0bba9f14b1c6f648c18f6780c83161ed3d98a97742621d274fd1f1de not found: ID does not exist" Oct 30 05:58:43.600763 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.600742 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b"] Oct 30 05:58:43.605546 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.605527 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-d4ccc5-9hf2b"] Oct 30 05:58:43.651526 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.651502 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:58:43.655474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.655456 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d4ccc5-dd9ff4874-vdtpf"] Oct 30 05:58:43.673780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.673759 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13896cfb-594f-4652-ac3e-6dc1940da08b" path="/var/lib/kubelet/pods/13896cfb-594f-4652-ac3e-6dc1940da08b/volumes" Oct 30 05:58:43.674274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:43.674259 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a3aa13d-2b73-40d8-a25f-0a03d87b2aca" path="/var/lib/kubelet/pods/3a3aa13d-2b73-40d8-a25f-0a03d87b2aca/volumes" Oct 30 05:58:44.372006 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:44.371971 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:58:44.408093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:44.408066 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:58:45.178071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:45.174911 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:58:45.207217 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:45.207186 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:58:46.782898 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:46.782864 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:58:46.783432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:46.783112 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" podUID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" containerName="maas-api" containerID="cri-o://3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d" gracePeriod=30 Oct 30 05:58:47.342099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.334494 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:58:47.342099 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.334771 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" podUID="80559662-d089-4324-898e-7563c22c4d4b" containerName="maas-api" containerID="cri-o://6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1" gracePeriod=30 Oct 30 05:58:47.583077 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.569219 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.635683 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\" (UniqueName: \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\") pod \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.635742 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\") pod \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.635848 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\") pod \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.636095 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\") pod \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\" (UID: \"1ad9a498-4870-4fd8-955b-d8af89bdeb97\") " Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.637097 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca" pod "1ad9a498-4870-4fd8-955b-d8af89bdeb97" (UID: "1ad9a498-4870-4fd8-955b-d8af89bdeb97"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.643781 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg" pod "1ad9a498-4870-4fd8-955b-d8af89bdeb97" (UID: "1ad9a498-4870-4fd8-955b-d8af89bdeb97"). InnerVolumeSpecName "kube-api-access-dhtwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.662597 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs" pod "1ad9a498-4870-4fd8-955b-d8af89bdeb97" (UID: "1ad9a498-4870-4fd8-955b-d8af89bdeb97"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:47.674534 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.665341 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls" pod "1ad9a498-4870-4fd8-955b-d8af89bdeb97" (UID: "1ad9a498-4870-4fd8-955b-d8af89bdeb97"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:47.702881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.702830 6171 generic.go:356] "Generic (PLEG): container finished" podID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" containerID="3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d" exitCode=0 Oct 30 05:58:47.704171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.703597 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" event={"ID":"1ad9a498-4870-4fd8-955b-d8af89bdeb97","Type":"ContainerDied","Data":"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d"} Oct 30 05:58:47.704171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.703658 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" event={"ID":"1ad9a498-4870-4fd8-955b-d8af89bdeb97","Type":"ContainerDied","Data":"a1e7b97c4b9af306f0ea766edbb847fc51526502382c09ae605e97855e6b7f2f"} Oct 30 05:58:47.704171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.703688 6171 scope.go:122] "RemoveContainer" containerID="3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d" Oct 30 05:58:47.704171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.703825 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74" Oct 30 05:58:47.737893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.737860 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:47.737893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.737888 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ad9a498-4870-4fd8-955b-d8af89bdeb97-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:47.738082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.737903 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhtwg\" (UniqueName: \"kubernetes.io/projected/1ad9a498-4870-4fd8-955b-d8af89bdeb97-kube-api-access-dhtwg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:47.738082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.737916 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ad9a498-4870-4fd8-955b-d8af89bdeb97-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:47.771500 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.771469 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:58:47.776330 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.776307 6171 scope.go:122] "RemoveContainer" containerID="3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d" Oct 30 05:58:47.776690 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.776670 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-3571f8-d6b59cd5d-tlf74"] Oct 30 05:58:47.776766 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:47.776736 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d\": container with ID starting with 3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d not found: ID does not exist" containerID="3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d" Oct 30 05:58:47.776806 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:47.776760 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d"} err="failed to get container status \"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d\": rpc error: code = NotFound desc = could not find container \"3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d\": container with ID starting with 3ee188545f96ac8235a4b51df53c97a02c3f58257a6813c2c62ed386b639175d not found: ID does not exist" Oct 30 05:58:48.286150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.286122 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:58:48.442686 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.442592 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\") pod \"80559662-d089-4324-898e-7563c22c4d4b\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " Oct 30 05:58:48.442859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.442795 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\") pod \"80559662-d089-4324-898e-7563c22c4d4b\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " Oct 30 05:58:48.442928 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.442886 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\" (UniqueName: \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\") pod \"80559662-d089-4324-898e-7563c22c4d4b\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " Oct 30 05:58:48.443102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.443077 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\") pod \"80559662-d089-4324-898e-7563c22c4d4b\" (UID: \"80559662-d089-4324-898e-7563c22c4d4b\") " Oct 30 05:58:48.443246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.443211 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca" pod "80559662-d089-4324-898e-7563c22c4d4b" (UID: "80559662-d089-4324-898e-7563c22c4d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:48.443663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.443647 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80559662-d089-4324-898e-7563c22c4d4b-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:48.445018 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.444982 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs" pod "80559662-d089-4324-898e-7563c22c4d4b" (UID: "80559662-d089-4324-898e-7563c22c4d4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:48.445415 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.445395 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9" pod "80559662-d089-4324-898e-7563c22c4d4b" (UID: "80559662-d089-4324-898e-7563c22c4d4b"). InnerVolumeSpecName "kube-api-access-bkdn9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:48.445682 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.445656 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls" pod "80559662-d089-4324-898e-7563c22c4d4b" (UID: "80559662-d089-4324-898e-7563c22c4d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:48.549078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.547140 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:48.549078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.547178 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80559662-d089-4324-898e-7563c22c4d4b-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:48.549078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.547195 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkdn9\" (UniqueName: \"kubernetes.io/projected/80559662-d089-4324-898e-7563c22c4d4b-kube-api-access-bkdn9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:48.715427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.715344 6171 generic.go:356] "Generic (PLEG): container finished" podID="80559662-d089-4324-898e-7563c22c4d4b" containerID="6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1" exitCode=0 Oct 30 05:58:48.715427 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.715388 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" Oct 30 05:58:48.715640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.715434 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" event={"ID":"80559662-d089-4324-898e-7563c22c4d4b","Type":"ContainerDied","Data":"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1"} Oct 30 05:58:48.715640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.715481 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl" event={"ID":"80559662-d089-4324-898e-7563c22c4d4b","Type":"ContainerDied","Data":"36aa26b88dfe35480599a96a76996ab40ad00e8e9ea6aafb07b7b8734558fb26"} Oct 30 05:58:48.715640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.715504 6171 scope.go:122] "RemoveContainer" containerID="6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1" Oct 30 05:58:48.727809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.727788 6171 scope.go:122] "RemoveContainer" containerID="6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1" Oct 30 05:58:48.728105 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:48.728083 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1\": container with ID starting with 6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1 not found: ID does not exist" containerID="6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1" Oct 30 05:58:48.728193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.728112 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1"} err="failed to get container status \"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1\": rpc error: code = NotFound desc = could not find container \"6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1\": container with ID starting with 6271f38fb876cfb2041beecf954f1e2e970867b01b3611d6c6c296de500952f1 not found: ID does not exist" Oct 30 05:58:48.788358 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.788338 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:58:48.807859 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:48.807833 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3efab09-84b4999bbb-qjfrl"] Oct 30 05:58:49.372660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.372613 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" podUID="7d719668-a732-47c3-80ac-98525d78bd2f" containerName="payload-processing" containerID="cri-o://1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec" gracePeriod=25 Oct 30 05:58:49.408597 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.408568 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" podUID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" containerName="payload-pre-processing" containerID="cri-o://c9e2d2bddc5e3c805de29c60d85f4a429bca3225ca272f50a5b751b87558aa40" gracePeriod=25 Oct 30 05:58:49.615024 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.614072 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:49.630114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.629849 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:49.681904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.681873 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" path="/var/lib/kubelet/pods/1ad9a498-4870-4fd8-955b-d8af89bdeb97/volumes" Oct 30 05:58:49.682609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.682587 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80559662-d089-4324-898e-7563c22c4d4b" path="/var/lib/kubelet/pods/80559662-d089-4324-898e-7563c22c4d4b/volumes" Oct 30 05:58:49.718171 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.718149 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:58:49.729620 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.729586 6171 generic.go:356] "Generic (PLEG): container finished" podID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" containerID="c9e2d2bddc5e3c805de29c60d85f4a429bca3225ca272f50a5b751b87558aa40" exitCode=0 Oct 30 05:58:49.729868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.729846 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" event={"ID":"3e158c7c-945b-4a2a-b184-ab2c2500bc42","Type":"ContainerDied","Data":"c9e2d2bddc5e3c805de29c60d85f4a429bca3225ca272f50a5b751b87558aa40"} Oct 30 05:58:49.731541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.731519 6171 generic.go:356] "Generic (PLEG): container finished" podID="7d719668-a732-47c3-80ac-98525d78bd2f" containerID="1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec" exitCode=0 Oct 30 05:58:49.731665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.731565 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" Oct 30 05:58:49.731665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.731595 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" event={"ID":"7d719668-a732-47c3-80ac-98525d78bd2f","Type":"ContainerDied","Data":"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec"} Oct 30 05:58:49.731665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.731639 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6" event={"ID":"7d719668-a732-47c3-80ac-98525d78bd2f","Type":"ContainerDied","Data":"79d93ef9c83a9f87af9f50402b0a7bdab68aaf23e63f81338cea6f7621c3c478"} Oct 30 05:58:49.731782 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.731669 6171 scope.go:122] "RemoveContainer" containerID="1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec" Oct 30 05:58:49.743233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.743215 6171 scope.go:122] "RemoveContainer" containerID="1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec" Oct 30 05:58:49.743527 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.743508 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 05:58:49.743626 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:49.743607 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec\": container with ID starting with 1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec not found: ID does not exist" containerID="1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec" Oct 30 05:58:49.743676 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.743640 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec"} err="failed to get container status \"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec\": rpc error: code = NotFound desc = could not find container \"1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec\": container with ID starting with 1840d1087c6257d6f26d7c7995254790547e3b0360926f8cf58bbcc99a31fcec not found: ID does not exist" Oct 30 05:58:49.781599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.781580 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:58:49.866733 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.866672 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\") pod \"7d719668-a732-47c3-80ac-98525d78bd2f\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " Oct 30 05:58:49.866988 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.866840 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\" (UniqueName: \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\") pod \"7d719668-a732-47c3-80ac-98525d78bd2f\" (UID: \"7d719668-a732-47c3-80ac-98525d78bd2f\") " Oct 30 05:58:49.867194 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.867167 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume" pod "7d719668-a732-47c3-80ac-98525d78bd2f" (UID: "7d719668-a732-47c3-80ac-98525d78bd2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:49.867716 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.867674 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d719668-a732-47c3-80ac-98525d78bd2f-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:49.869312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.869287 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv" pod "7d719668-a732-47c3-80ac-98525d78bd2f" (UID: "7d719668-a732-47c3-80ac-98525d78bd2f"). InnerVolumeSpecName "kube-api-access-r4rpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:49.968745 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.968664 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\" (UniqueName: \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\") pod \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " Oct 30 05:58:49.968906 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.968893 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\") pod \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\" (UID: \"3e158c7c-945b-4a2a-b184-ab2c2500bc42\") " Oct 30 05:58:49.969286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.969261 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume" pod "3e158c7c-945b-4a2a-b184-ab2c2500bc42" (UID: "3e158c7c-945b-4a2a-b184-ab2c2500bc42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:49.969483 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.969469 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r4rpv\" (UniqueName: \"kubernetes.io/projected/7d719668-a732-47c3-80ac-98525d78bd2f-kube-api-access-r4rpv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:49.969532 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.969488 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e158c7c-945b-4a2a-b184-ab2c2500bc42-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:49.970818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:49.970792 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l" pod "3e158c7c-945b-4a2a-b184-ab2c2500bc42" (UID: "3e158c7c-945b-4a2a-b184-ab2c2500bc42"). InnerVolumeSpecName "kube-api-access-kct6l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:50.071476 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.071448 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kct6l\" (UniqueName: \"kubernetes.io/projected/3e158c7c-945b-4a2a-b184-ab2c2500bc42-kube-api-access-kct6l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.087305 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.087279 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:58:50.094673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.094637 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a3efab09-6f7cc965b5-tjzb6"] Oct 30 05:58:50.176410 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.176374 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" podUID="fa9900f8-0bd5-4e87-9947-99575bf2e077" containerName="payload-processing" containerID="cri-o://2100670daaaf02e22db323cc82687b07c604f8eb7d804dc91602c4c9bff03535" gracePeriod=25 Oct 30 05:58:50.210445 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.210403 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" podUID="730b950b-5e83-4b84-b92b-ebb3f45560af" containerName="payload-pre-processing" containerID="cri-o://8dd5394836ed3dc1e90ef821a5aa8e1e22e94a105e12cdc934cdfda1e6772b20" gracePeriod=25 Oct 30 05:58:50.389191 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.385011 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 05:58:50.784772 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.784715 6171 generic.go:356] "Generic (PLEG): container finished" podID="fa9900f8-0bd5-4e87-9947-99575bf2e077" containerID="2100670daaaf02e22db323cc82687b07c604f8eb7d804dc91602c4c9bff03535" exitCode=0 Oct 30 05:58:50.784944 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.784874 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" event={"ID":"fa9900f8-0bd5-4e87-9947-99575bf2e077","Type":"ContainerDied","Data":"2100670daaaf02e22db323cc82687b07c604f8eb7d804dc91602c4c9bff03535"} Oct 30 05:58:50.785022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.784950 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" event={"ID":"fa9900f8-0bd5-4e87-9947-99575bf2e077","Type":"ContainerDied","Data":"0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26"} Oct 30 05:58:50.785022 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.784991 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c90134fd3084a1267a948a784901175420fbd643ccddd5c78b1997f5f849c26" Oct 30 05:58:50.787088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.787036 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" event={"ID":"3e158c7c-945b-4a2a-b184-ab2c2500bc42","Type":"ContainerDied","Data":"a777faca44d92935ce3b02f351b0285a490bc6612c9886b0624411f0101d8717"} Oct 30 05:58:50.787224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.787108 6171 scope.go:122] "RemoveContainer" containerID="c9e2d2bddc5e3c805de29c60d85f4a429bca3225ca272f50a5b751b87558aa40" Oct 30 05:58:50.787224 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.787209 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5" Oct 30 05:58:50.795854 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.795833 6171 generic.go:356] "Generic (PLEG): container finished" podID="730b950b-5e83-4b84-b92b-ebb3f45560af" containerID="8dd5394836ed3dc1e90ef821a5aa8e1e22e94a105e12cdc934cdfda1e6772b20" exitCode=0 Oct 30 05:58:50.796003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.795984 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" event={"ID":"730b950b-5e83-4b84-b92b-ebb3f45560af","Type":"ContainerDied","Data":"8dd5394836ed3dc1e90ef821a5aa8e1e22e94a105e12cdc934cdfda1e6772b20"} Oct 30 05:58:50.801775 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.801756 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:58:50.852832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.852807 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:58:50.858516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.858494 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a3efab09-7849ccc8c5-ccwp5"] Oct 30 05:58:50.869838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.869818 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:58:50.918583 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.918554 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\") pod \"fa9900f8-0bd5-4e87-9947-99575bf2e077\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " Oct 30 05:58:50.918780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.918760 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\") pod \"730b950b-5e83-4b84-b92b-ebb3f45560af\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " Oct 30 05:58:50.918904 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.918883 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume" pod "fa9900f8-0bd5-4e87-9947-99575bf2e077" (UID: "fa9900f8-0bd5-4e87-9947-99575bf2e077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:50.918974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.918889 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\" (UniqueName: \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\") pod \"730b950b-5e83-4b84-b92b-ebb3f45560af\" (UID: \"730b950b-5e83-4b84-b92b-ebb3f45560af\") " Oct 30 05:58:50.918974 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.918962 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\" (UniqueName: \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\") pod \"fa9900f8-0bd5-4e87-9947-99575bf2e077\" (UID: \"fa9900f8-0bd5-4e87-9947-99575bf2e077\") " Oct 30 05:58:50.919221 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.919198 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume" pod "730b950b-5e83-4b84-b92b-ebb3f45560af" (UID: "730b950b-5e83-4b84-b92b-ebb3f45560af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:50.919809 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.919786 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa9900f8-0bd5-4e87-9947-99575bf2e077-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.919935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.919814 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/730b950b-5e83-4b84-b92b-ebb3f45560af-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:50.921318 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.921290 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd" pod "730b950b-5e83-4b84-b92b-ebb3f45560af" (UID: "730b950b-5e83-4b84-b92b-ebb3f45560af"). InnerVolumeSpecName "kube-api-access-l5fmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:50.921624 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:50.921601 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9" pod "fa9900f8-0bd5-4e87-9947-99575bf2e077" (UID: "fa9900f8-0bd5-4e87-9947-99575bf2e077"). InnerVolumeSpecName "kube-api-access-h9bs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:51.021360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.021325 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5fmd\" (UniqueName: \"kubernetes.io/projected/730b950b-5e83-4b84-b92b-ebb3f45560af-kube-api-access-l5fmd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:51.021360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.021353 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9bs9\" (UniqueName: \"kubernetes.io/projected/fa9900f8-0bd5-4e87-9947-99575bf2e077-kube-api-access-h9bs9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:51.753595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.737781 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" path="/var/lib/kubelet/pods/3e158c7c-945b-4a2a-b184-ab2c2500bc42/volumes" Oct 30 05:58:51.757262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.757232 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d719668-a732-47c3-80ac-98525d78bd2f" path="/var/lib/kubelet/pods/7d719668-a732-47c3-80ac-98525d78bd2f/volumes" Oct 30 05:58:51.887972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.887934 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z" Oct 30 05:58:51.888154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.888129 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" event={"ID":"730b950b-5e83-4b84-b92b-ebb3f45560af","Type":"ContainerDied","Data":"17d3e8a9f02cdb21eaa9ccf06b62f8cee7abd0c98d8335252ce6c4b9a688b60b"} Oct 30 05:58:51.888236 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.888204 6171 scope.go:122] "RemoveContainer" containerID="8dd5394836ed3dc1e90ef821a5aa8e1e22e94a105e12cdc934cdfda1e6772b20" Oct 30 05:58:51.888361 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.888340 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m" Oct 30 05:58:51.976070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:51.975018 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:58:52.012108 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:52.010861 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1b2243ef-7955fbbb49-7wv7z"] Oct 30 05:58:52.017342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:52.017315 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:58:52.034093 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:52.034038 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-1b2243ef-7bb8d879cf-7k92m"] Oct 30 05:58:52.432206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:52.430623 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:58:52.432206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:52.431419 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" containerID="cri-o://19da64d8eb1f7d43928796804247eb1373d4f79dfc298303e2a1ce32194cb919" gracePeriod=30 Oct 30 05:58:53.673663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:53.673634 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="730b950b-5e83-4b84-b92b-ebb3f45560af" path="/var/lib/kubelet/pods/730b950b-5e83-4b84-b92b-ebb3f45560af/volumes" Oct 30 05:58:53.674079 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:53.674060 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa9900f8-0bd5-4e87-9947-99575bf2e077" path="/var/lib/kubelet/pods/fa9900f8-0bd5-4e87-9947-99575bf2e077/volumes" Oct 30 05:58:55.333123 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:55.333083 6171 patch_prober.go:29] interesting pod/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:58:55.333599 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:55.333157 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:58:56.543086 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:58:56.534304 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:58:58.000575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.000325 6171 generic.go:356] "Generic (PLEG): container finished" podID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerID="19da64d8eb1f7d43928796804247eb1373d4f79dfc298303e2a1ce32194cb919" exitCode=0 Oct 30 05:58:58.000575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.000431 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerDied","Data":"19da64d8eb1f7d43928796804247eb1373d4f79dfc298303e2a1ce32194cb919"} Oct 30 05:58:58.000575 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.000487 6171 scope.go:122] "RemoveContainer" containerID="e91641a98e117442b4a47c23e4fe85a27662f968c13f9acffb125b310691be1b" Oct 30 05:58:58.601621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.601528 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:58:58.692072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.691167 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.692072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.691236 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.692072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.691390 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.692072 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.691428 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.696542 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.696500 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:58.711270 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.711220 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.817639 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.817754 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819124 6171 cpu_manager.go:414] "containerMap: removing container" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819145 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819162 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7d719668-a732-47c3-80ac-98525d78bd2f" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819171 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d719668-a732-47c3-80ac-98525d78bd2f" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819183 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819190 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819207 6171 cpu_manager.go:414] "containerMap: removing container" podUID="730b950b-5e83-4b84-b92b-ebb3f45560af" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819214 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="730b950b-5e83-4b84-b92b-ebb3f45560af" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819250 6171 cpu_manager.go:414] "containerMap: removing container" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819257 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819270 6171 cpu_manager.go:414] "containerMap: removing container" podUID="80559662-d089-4324-898e-7563c22c4d4b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819278 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80559662-d089-4324-898e-7563c22c4d4b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819309 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819317 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819374 6171 cpu_manager.go:414] "containerMap: removing container" podUID="13896cfb-594f-4652-ac3e-6dc1940da08b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819381 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13896cfb-594f-4652-ac3e-6dc1940da08b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819392 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fa9900f8-0bd5-4e87-9947-99575bf2e077" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819400 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa9900f8-0bd5-4e87-9947-99575bf2e077" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819583 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="13896cfb-594f-4652-ac3e-6dc1940da08b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819596 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa9900f8-0bd5-4e87-9947-99575bf2e077" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819610 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819619 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d719668-a732-47c3-80ac-98525d78bd2f" containerName="payload-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819630 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ad9a498-4870-4fd8-955b-d8af89bdeb97" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819640 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="730b950b-5e83-4b84-b92b-ebb3f45560af" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819650 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" containerName="istio-proxy" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819660 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="80559662-d089-4324-898e-7563c22c4d4b" containerName="maas-api" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.819673 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e158c7c-945b-4a2a-b184-ab2c2500bc42" containerName="payload-pre-processing" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.831097 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.840853 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.840920 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.840992 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.841055 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.841091 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\") pod \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\" (UID: \"56e9e3ff-400c-4e70-90d4-7efdacaf70ea\") " Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.842025 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.842073 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.842089 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:58.842155 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.842149 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:58.862771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.862087 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:58.862771 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.862443 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:58.870432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.867557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "kube-api-access-cwr88". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:58.870432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.867998 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:58.870432 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.868704 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket" pod "56e9e3ff-400c-4e70-90d4-7efdacaf70ea" (UID: "56e9e3ff-400c-4e70-90d4-7efdacaf70ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:58.972837 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.972792 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 05:58:58.973034 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.972940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:58.975802 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.975774 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:59.006132 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.006101 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:59.016126 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.016099 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:59.016307 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.016296 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:59.016417 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.016405 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwr88\" (UniqueName: \"kubernetes.io/projected/56e9e3ff-400c-4e70-90d4-7efdacaf70ea-kube-api-access-cwr88\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:58:59.018868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:58.992522 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-84dc9d-openshift-default-dockercfg-9gjw7\"" Oct 30 05:58:59.031088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.030852 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" event={"ID":"56e9e3ff-400c-4e70-90d4-7efdacaf70ea","Type":"ContainerDied","Data":"7bc2ff9e23da796b53ba367bb24939a2314b81acbb73108413f6ee6f245bf8dc"} Oct 30 05:58:59.031088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.030917 6171 scope.go:122] "RemoveContainer" containerID="19da64d8eb1f7d43928796804247eb1373d4f79dfc298303e2a1ce32194cb919" Oct 30 05:58:59.031088 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.030871 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq" Oct 30 05:58:59.118138 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118001 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118210 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118323 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118420 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118402 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118474 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118419 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:58:59.118519 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118476 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118574 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118627 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khqm6\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118752 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118732 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.118820 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.118778 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.123375 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.123349 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d4ccc5-openshift-default-646f9fd445-sjcsq"] Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220218 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220330 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220439 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220517 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220562 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.231070 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.220644 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khqm6\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.262316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.258669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.262316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.258744 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.262316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.258827 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.262316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.241410 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.262316 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.229542 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.282105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.232340 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.282105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.240969 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.282105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.279927 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.282105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.280152 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.329071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.322958 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khqm6\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.331531 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.331477 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.334586 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.334554 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\") pod \"e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.631623 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.631589 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:58:59.675075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.675024 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56e9e3ff-400c-4e70-90d4-7efdacaf70ea" path="/var/lib/kubelet/pods/56e9e3ff-400c-4e70-90d4-7efdacaf70ea/volumes" Oct 30 05:58:59.828818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.828734 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:58:59.829071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:58:59.829017 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" podUID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" containerName="maas-api" containerID="cri-o://e97bd902ba00fb7b15727eef501d8cc8cf96f8985e2eb98019e31358b7706dae" gracePeriod=30 Oct 30 05:59:00.002304 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.002279 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 05:59:00.005844 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:00.005809 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc8eba30_ea37_4750_939f_dd56ff6d6e9e.slice/crio-8891436393b19288f202b042422f1ca14ccee79e2c41c91d4a9ca096ad6de79b WatchSource:0}: Error finding container 8891436393b19288f202b042422f1ca14ccee79e2c41c91d4a9ca096ad6de79b: Status 404 returned error can't find the container with id 8891436393b19288f202b042422f1ca14ccee79e2c41c91d4a9ca096ad6de79b Oct 30 05:59:00.009038 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.009000 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:00.009450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.009090 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:00.009450 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.009123 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:00.056128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.055771 6171 generic.go:356] "Generic (PLEG): container finished" podID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" containerID="e97bd902ba00fb7b15727eef501d8cc8cf96f8985e2eb98019e31358b7706dae" exitCode=0 Oct 30 05:59:00.056128 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.056028 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" event={"ID":"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2","Type":"ContainerDied","Data":"e97bd902ba00fb7b15727eef501d8cc8cf96f8985e2eb98019e31358b7706dae"} Oct 30 05:59:00.061908 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.061691 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerStarted","Data":"8891436393b19288f202b042422f1ca14ccee79e2c41c91d4a9ca096ad6de79b"} Oct 30 05:59:00.275993 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.275960 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:59:00.389154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.382854 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\" (UniqueName: \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\") pod \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " Oct 30 05:59:00.389154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.383021 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\") pod \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " Oct 30 05:59:00.389154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.383086 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\") pod \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " Oct 30 05:59:00.389154 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.383236 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\") pod \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\" (UID: \"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2\") " Oct 30 05:59:00.502118 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.496059 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl" pod "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" (UID: "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2"). InnerVolumeSpecName "kube-api-access-x26kl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:00.537436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.537398 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca" pod "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" (UID: "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:00.558110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.546171 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs" pod "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" (UID: "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:00.558110 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.546254 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls" pod "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" (UID: "df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:00.622114 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.599991 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:00.629338 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.629305 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:00.629573 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.629549 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:00.650169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:00.650093 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26kl\" (UniqueName: \"kubernetes.io/projected/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2-kube-api-access-x26kl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:01.084090 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.084053 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerStarted","Data":"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449"} Oct 30 05:59:01.086103 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.086037 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" Oct 30 05:59:01.086226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.086108 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2" event={"ID":"df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2","Type":"ContainerDied","Data":"46ba5f0b7f5e83db4d60f3761815c6b9496e6ccbfbec14bd512dc46b351f02a3"} Oct 30 05:59:01.086226 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.086179 6171 scope.go:122] "RemoveContainer" containerID="e97bd902ba00fb7b15727eef501d8cc8cf96f8985e2eb98019e31358b7706dae" Oct 30 05:59:01.193773 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.193530 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podStartSLOduration=3.193507494 podStartE2EDuration="3.193507494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:01.174523042 +0000 UTC m=+1599.614900269" watchObservedRunningTime="2026-10-30 05:59:01.193507494 +0000 UTC m=+1599.633884689" Oct 30 05:59:01.230959 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.230926 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:59:01.237105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.237078 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1b2243ef-54bff4576-f6bz2"] Oct 30 05:59:01.631934 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.631903 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:59:01.637402 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.637376 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:59:01.675886 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:01.675846 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" path="/var/lib/kubelet/pods/df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2/volumes" Oct 30 05:59:02.107085 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:02.107037 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:59:02.108170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:02.108149 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 05:59:03.659735 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:03.659699 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:59:05.164462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.163211 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:59:05.164462 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.163281 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:59:05.217615 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.217579 6171 generic.go:356] "Generic (PLEG): container finished" podID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerID="9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83" exitCode=0 Oct 30 05:59:05.217796 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.217665 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerDied","Data":"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83"} Oct 30 05:59:05.218280 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.218263 6171 scope.go:122] "RemoveContainer" containerID="9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83" Oct 30 05:59:05.220242 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.220212 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:05.220337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.220281 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:05.220337 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.220314 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:05.537838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.537792 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 05:59:05.538661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.538640 6171 cpu_manager.go:414] "containerMap: removing container" podUID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" containerName="maas-api" Oct 30 05:59:05.538661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.538656 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" containerName="maas-api" Oct 30 05:59:05.538868 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.538804 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="df8bc21c-ea4e-49ab-a1e6-45d52a27b2a2" containerName="maas-api" Oct 30 05:59:05.542767 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.542741 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.562760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.562732 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-84dc9d\"" Oct 30 05:59:05.566609 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.566581 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 05:59:05.587538 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.587504 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.587736 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.587702 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs698\" (UniqueName: \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.622517 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.622489 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 05:59:05.627276 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.627256 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.647101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.647069 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 05:59:05.652739 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.652709 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-84dc9d-dockercfg-pq7wh\"" Oct 30 05:59:05.690158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.690124 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs698\" (UniqueName: \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.690359 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.690226 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4h48l\" (UniqueName: \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.690359 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.690340 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.690503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.690415 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.691167 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.691143 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.716621 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.716587 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs698\" (UniqueName: \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\") pod \"payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:05.802129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.800339 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4h48l\" (UniqueName: \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.802129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.800490 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.802129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.801594 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.871078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.844803 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4h48l\" (UniqueName: \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\") pod \"payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:05.876436 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:05.874691 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:06.009807 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:06.009774 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:06.246246 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:06.246211 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerStarted","Data":"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf"} Oct 30 05:59:06.417109 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:06.416985 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 05:59:06.426308 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:06.426274 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0573b9c_6a49_48fc_9f23_8e11e909d758.slice/crio-e5b67ee8c0b3c1b7df21691258fe7e232c72315931e8c8b7013677626633d19e WatchSource:0}: Error finding container e5b67ee8c0b3c1b7df21691258fe7e232c72315931e8c8b7013677626633d19e: Status 404 returned error can't find the container with id e5b67ee8c0b3c1b7df21691258fe7e232c72315931e8c8b7013677626633d19e Oct 30 05:59:06.446661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:06.446632 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 05:59:06.450608 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:06.450577 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd144f7a0_4695_49cf_bce2_711fc2f550ea.slice/crio-753b6f39d2b853470f82a3e1c56b0a9180316be6174e6e58024cf7df2ebf7f10 WatchSource:0}: Error finding container 753b6f39d2b853470f82a3e1c56b0a9180316be6174e6e58024cf7df2ebf7f10: Status 404 returned error can't find the container with id 753b6f39d2b853470f82a3e1c56b0a9180316be6174e6e58024cf7df2ebf7f10 Oct 30 05:59:06.464058 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:06.464014 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:59:07.061233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.061202 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:59:07.061440 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.061418 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 05:59:07.061515 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.061493 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 05:59:07.257274 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.257235 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" event={"ID":"d144f7a0-4695-49cf-bce2-711fc2f550ea","Type":"ContainerStarted","Data":"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8"} Oct 30 05:59:07.257732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.257300 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" event={"ID":"d144f7a0-4695-49cf-bce2-711fc2f550ea","Type":"ContainerStarted","Data":"753b6f39d2b853470f82a3e1c56b0a9180316be6174e6e58024cf7df2ebf7f10"} Oct 30 05:59:07.257732 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.257328 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:07.260890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.259370 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" event={"ID":"d0573b9c-6a49-48fc-9f23-8e11e909d758","Type":"ContainerStarted","Data":"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de"} Oct 30 05:59:07.260890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.259423 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" event={"ID":"d0573b9c-6a49-48fc-9f23-8e11e909d758","Type":"ContainerStarted","Data":"e5b67ee8c0b3c1b7df21691258fe7e232c72315931e8c8b7013677626633d19e"} Oct 30 05:59:07.313349 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.313202 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" podStartSLOduration=2.313180403 podStartE2EDuration="2.313180403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:07.312723827 +0000 UTC m=+1605.753101033" watchObservedRunningTime="2026-10-30 05:59:07.313180403 +0000 UTC m=+1605.753557597" Oct 30 05:59:07.373418 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.373355 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" podStartSLOduration=2.373336177 podStartE2EDuration="2.373336177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:07.370787959 +0000 UTC m=+1605.811165152" watchObservedRunningTime="2026-10-30 05:59:07.373336177 +0000 UTC m=+1605.813713371" Oct 30 05:59:07.948964 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:07.948928 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:59:08.061746 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:08.061695 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 05:59:08.061929 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:08.061765 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 05:59:08.360953 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:08.360929 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:08.665133 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:08.664811 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" podUID="3b964009-2b1c-4739-9ea3-3205beb9e37e" containerName="payload-pre-processing" containerID="cri-o://3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4" gracePeriod=25 Oct 30 05:59:09.063282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.062350 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 05:59:09.063282 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.062447 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 05:59:09.234262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.234234 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:59:09.258255 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.258222 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\") pod \"3b964009-2b1c-4739-9ea3-3205beb9e37e\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " Oct 30 05:59:09.258442 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.258324 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\" (UniqueName: \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\") pod \"3b964009-2b1c-4739-9ea3-3205beb9e37e\" (UID: \"3b964009-2b1c-4739-9ea3-3205beb9e37e\") " Oct 30 05:59:09.258619 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.258595 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume" pod "3b964009-2b1c-4739-9ea3-3205beb9e37e" (UID: "3b964009-2b1c-4739-9ea3-3205beb9e37e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:09.259102 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.259083 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b964009-2b1c-4739-9ea3-3205beb9e37e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:09.260836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.260815 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8" pod "3b964009-2b1c-4739-9ea3-3205beb9e37e" (UID: "3b964009-2b1c-4739-9ea3-3205beb9e37e"). InnerVolumeSpecName "kube-api-access-f85w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:09.361232 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.361200 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f85w8\" (UniqueName: \"kubernetes.io/projected/3b964009-2b1c-4739-9ea3-3205beb9e37e-kube-api-access-f85w8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:09.372116 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.372022 6171 generic.go:356] "Generic (PLEG): container finished" podID="3b964009-2b1c-4739-9ea3-3205beb9e37e" containerID="3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4" exitCode=0 Oct 30 05:59:09.372268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.372122 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" Oct 30 05:59:09.372268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.372190 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" event={"ID":"3b964009-2b1c-4739-9ea3-3205beb9e37e","Type":"ContainerDied","Data":"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4"} Oct 30 05:59:09.372268 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.372239 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn" event={"ID":"3b964009-2b1c-4739-9ea3-3205beb9e37e","Type":"ContainerDied","Data":"72b5bddc9b8610d59734bc57ccf1754a625171a8032c45f9a25821df4a8cc1eb"} Oct 30 05:59:09.372475 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.372263 6171 scope.go:122] "RemoveContainer" containerID="3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4" Oct 30 05:59:09.396369 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.395296 6171 scope.go:122] "RemoveContainer" containerID="3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4" Oct 30 05:59:09.397350 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:09.397308 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4\": container with ID starting with 3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4 not found: ID does not exist" containerID="3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4" Oct 30 05:59:09.397550 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.397498 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4"} err="failed to get container status \"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4\": rpc error: code = NotFound desc = could not find container \"3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4\": container with ID starting with 3d57f342ef423e35b2c57c9bf413bf27faa0bc8cdd1cbb2527cddcffdd74a6b4 not found: ID does not exist" Oct 30 05:59:09.439990 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.439958 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:59:09.444074 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.444038 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-b202a0-c64fdf9d7-n5hqn"] Oct 30 05:59:09.673912 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:09.673835 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b964009-2b1c-4739-9ea3-3205beb9e37e" path="/var/lib/kubelet/pods/3b964009-2b1c-4739-9ea3-3205beb9e37e/volumes" Oct 30 05:59:10.061611 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:10.061575 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 05:59:10.061770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:10.061628 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 05:59:11.061751 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:11.061719 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" start-of-body= Oct 30 05:59:11.062196 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:11.061784 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.201:15021/healthz/ready\": dial tcp 10.133.0.201:15021: connect: connection refused" Oct 30 05:59:12.154790 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.154740 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:59:12.155800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.155544 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:59:12.156760 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.156739 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 05:59:12.898656 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.898611 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 05:59:12.899616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.899593 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3b964009-2b1c-4739-9ea3-3205beb9e37e" containerName="payload-pre-processing" Oct 30 05:59:12.899616 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.899617 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b964009-2b1c-4739-9ea3-3205beb9e37e" containerName="payload-pre-processing" Oct 30 05:59:12.899862 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.899780 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b964009-2b1c-4739-9ea3-3205beb9e37e" containerName="payload-pre-processing" Oct 30 05:59:12.903798 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.903772 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.934648 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.934003 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 05:59:12.941903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.941883 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-de7242\"/\"openshift-service-ca.crt\"" Oct 30 05:59:12.941903 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.941892 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-de7242\"/\"kube-root-ca.crt\"" Oct 30 05:59:12.942107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.941942 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-de7242\"/\"default-dockercfg-rwfnv\"" Oct 30 05:59:12.942107 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.941955 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-de7242\"/\"test-model-de7242-kserve-self-signed-certs\"" Oct 30 05:59:12.950186 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.950154 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" podUID="76870fdd-4e06-4fcd-ac32-2c9689140113" containerName="payload-pre-processing" containerID="cri-o://83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118" gracePeriod=25 Oct 30 05:59:12.956890 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.956743 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.957026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.956910 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.957026 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.956958 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfb2\" (UniqueName: \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.957914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.957091 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.957914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.957306 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:12.957914 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:12.957333 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059275 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059380 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059397 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059494 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059482 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059561 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059696 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059618 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfb2\" (UniqueName: \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059814 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059760 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.059871 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.059814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.060162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.060137 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.061801 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.061781 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.061889 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.061801 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.085105 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.085037 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfb2\" (UniqueName: \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\") pod \"test-model-de7242-kserve-c9bf79595-bfl2f\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.258127 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.228823 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:13.432202 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.432173 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:59:13.481895 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.481752 6171 generic.go:356] "Generic (PLEG): container finished" podID="76870fdd-4e06-4fcd-ac32-2c9689140113" containerID="83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118" exitCode=0 Oct 30 05:59:13.483600 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.483571 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" Oct 30 05:59:13.484027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.483999 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" event={"ID":"76870fdd-4e06-4fcd-ac32-2c9689140113","Type":"ContainerDied","Data":"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118"} Oct 30 05:59:13.484225 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.484185 6171 scope.go:122] "RemoveContainer" containerID="83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118" Oct 30 05:59:13.485014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.484077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm" event={"ID":"76870fdd-4e06-4fcd-ac32-2c9689140113","Type":"ContainerDied","Data":"08a21dd42b96601ca907fcb42088da0c53460781d5e64684e12a44e62f314231"} Oct 30 05:59:13.497253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.496539 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\") pod \"76870fdd-4e06-4fcd-ac32-2c9689140113\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " Oct 30 05:59:13.497253 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.496786 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\" (UniqueName: \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\") pod \"76870fdd-4e06-4fcd-ac32-2c9689140113\" (UID: \"76870fdd-4e06-4fcd-ac32-2c9689140113\") " Oct 30 05:59:13.498351 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.498316 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume" pod "76870fdd-4e06-4fcd-ac32-2c9689140113" (UID: "76870fdd-4e06-4fcd-ac32-2c9689140113"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:13.504250 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.502988 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8" pod "76870fdd-4e06-4fcd-ac32-2c9689140113" (UID: "76870fdd-4e06-4fcd-ac32-2c9689140113"). InnerVolumeSpecName "kube-api-access-q88p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:13.510884 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.510745 6171 scope.go:122] "RemoveContainer" containerID="83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118" Oct 30 05:59:13.511275 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:13.511243 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118\": container with ID starting with 83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118 not found: ID does not exist" containerID="83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118" Oct 30 05:59:13.511377 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.511285 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118"} err="failed to get container status \"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118\": rpc error: code = NotFound desc = could not find container \"83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118\": container with ID starting with 83544715df95b847afcee76f321fbf084be8995f5daf46699d40b9954524c118 not found: ID does not exist" Oct 30 05:59:13.599378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.599335 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q88p8\" (UniqueName: \"kubernetes.io/projected/76870fdd-4e06-4fcd-ac32-2c9689140113-kube-api-access-q88p8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.599378 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.599371 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76870fdd-4e06-4fcd-ac32-2c9689140113-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:13.765942 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.765917 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 05:59:13.767441 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:13.767411 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7264c9e4_2f72_4cac_b559_cdc96c372d24.slice/crio-3cf1ac49d22027817802c99876db53de33f78b2859dbbe3d51db9f42ab00c562 WatchSource:0}: Error finding container 3cf1ac49d22027817802c99876db53de33f78b2859dbbe3d51db9f42ab00c562: Status 404 returned error can't find the container with id 3cf1ac49d22027817802c99876db53de33f78b2859dbbe3d51db9f42ab00c562 Oct 30 05:59:13.834780 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.834750 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:59:13.839412 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:13.839311 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4fc75f-64b8cb944c-hrqbm"] Oct 30 05:59:14.494632 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:14.494596 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" event={"ID":"7264c9e4-2f72-4cac-b559-cdc96c372d24","Type":"ContainerStarted","Data":"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26"} Oct 30 05:59:14.495140 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:14.494642 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" event={"ID":"7264c9e4-2f72-4cac-b559-cdc96c372d24","Type":"ContainerStarted","Data":"3cf1ac49d22027817802c99876db53de33f78b2859dbbe3d51db9f42ab00c562"} Oct 30 05:59:14.543262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:14.543208 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podStartSLOduration=2.543188061 podStartE2EDuration="2.543188061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:14.539742668 +0000 UTC m=+1612.980119860" watchObservedRunningTime="2026-10-30 05:59:14.543188061 +0000 UTC m=+1612.983565255" Oct 30 05:59:15.233541 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:15.233510 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:15.676333 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:15.676253 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76870fdd-4e06-4fcd-ac32-2c9689140113" path="/var/lib/kubelet/pods/76870fdd-4e06-4fcd-ac32-2c9689140113/volumes" Oct 30 05:59:18.362738 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:18.362709 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 05:59:19.406198 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:19.406155 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 05:59:21.570069 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:21.564202 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" containerID="cri-o://bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b" gracePeriod=45 Oct 30 05:59:21.910530 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:21.910504 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:59:22.007311 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007274 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007348 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007433 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007507 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007496 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007577 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\" (UniqueName: \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007695 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007623 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\") pod \"5c248106-d4bc-4ede-ba9f-552da4945b3c\" (UID: \"5c248106-d4bc-4ede-ba9f-552da4945b3c\") " Oct 30 05:59:22.007973 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.007943 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:22.008201 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.008112 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:22.008313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.008286 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:22.008657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.008632 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.008845 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.008830 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.008956 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.008945 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.010791 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.010753 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:22.010895 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.010802 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:22.011961 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.011926 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d" pod "5c248106-d4bc-4ede-ba9f-552da4945b3c" (UID: "5c248106-d4bc-4ede-ba9f-552da4945b3c"). InnerVolumeSpecName "kube-api-access-bk84d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:22.028617 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.028593 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp"] Oct 30 05:59:22.029935 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.029915 6171 cpu_manager.go:414] "containerMap: removing container" podUID="76870fdd-4e06-4fcd-ac32-2c9689140113" containerName="payload-pre-processing" Oct 30 05:59:22.030014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.029938 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76870fdd-4e06-4fcd-ac32-2c9689140113" containerName="payload-pre-processing" Oct 30 05:59:22.030014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.029966 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" Oct 30 05:59:22.030014 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.029974 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" Oct 30 05:59:22.030220 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.030206 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="76870fdd-4e06-4fcd-ac32-2c9689140113" containerName="payload-pre-processing" Oct 30 05:59:22.030256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.030226 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerName="main" Oct 30 05:59:22.034960 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.034944 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.055096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.055064 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp"] Oct 30 05:59:22.056256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.055611 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:59:22.056256 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.055677 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:59:22.110151 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110113 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.110353 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110296 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.110539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110513 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdwvg\" (UniqueName: \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.110794 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110773 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c248106-d4bc-4ede-ba9f-552da4945b3c-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.110872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110799 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bk84d\" (UniqueName: \"kubernetes.io/projected/5c248106-d4bc-4ede-ba9f-552da4945b3c-kube-api-access-bk84d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.110872 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.110815 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c248106-d4bc-4ede-ba9f-552da4945b3c-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:22.212539 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.212456 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdwvg\" (UniqueName: \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.212688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.212615 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.212737 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.212692 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.213278 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.213256 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.215172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.215151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.238893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.238863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdwvg\" (UniqueName: \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\") pod \"maas-api-revoke-sub-db02f378-2kmsp\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.379553 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.379523 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:22.645651 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.645616 6171 generic.go:356] "Generic (PLEG): container finished" podID="5c248106-d4bc-4ede-ba9f-552da4945b3c" containerID="bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b" exitCode=0 Oct 30 05:59:22.646158 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.645669 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" event={"ID":"5c248106-d4bc-4ede-ba9f-552da4945b3c","Type":"ContainerDied","Data":"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b"} Oct 30 05:59:22.646313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.646178 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" event={"ID":"5c248106-d4bc-4ede-ba9f-552da4945b3c","Type":"ContainerDied","Data":"db227eaaed37595d1b833931f26863fa81a68e90f93d73b358f8800122ea262d"} Oct 30 05:59:22.646313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.645742 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4" Oct 30 05:59:22.646313 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.646205 6171 scope.go:122] "RemoveContainer" containerID="bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b" Oct 30 05:59:22.665479 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.665457 6171 scope.go:122] "RemoveContainer" containerID="bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b" Oct 30 05:59:22.665768 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:22.665740 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b\": container with ID starting with bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b not found: ID does not exist" containerID="bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b" Oct 30 05:59:22.665870 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.665775 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b"} err="failed to get container status \"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b\": rpc error: code = NotFound desc = could not find container \"bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b\": container with ID starting with bdecdcfdd231e4c450d2335ddc96c1625d7cf8e05974dc0debcc60fd8ef8d99b not found: ID does not exist" Oct 30 05:59:22.690817 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:22.690788 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc26d67_28d8_41ee_a77f_11d2ab469596.slice/crio-982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0 WatchSource:0}: Error finding container 982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0: Status 404 returned error can't find the container with id 982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0 Oct 30 05:59:22.692257 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.692237 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp"] Oct 30 05:59:22.709321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.709299 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:59:22.714233 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:22.714210 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-1b2243ef/e2e-lifecycle-model-1b2243ef-kserve-9cccb7cd9-q5kb4"] Oct 30 05:59:23.233264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.233234 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:23.245652 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.245629 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:23.656204 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.656167 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" event={"ID":"cfc26d67-28d8-41ee-a77f-11d2ab469596","Type":"ContainerStarted","Data":"3b99719f3da86f9fd2eba03443dfdde47b85b5bf73dfda28f6052d7df7f8306d"} Oct 30 05:59:23.656654 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.656228 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" event={"ID":"cfc26d67-28d8-41ee-a77f-11d2ab469596","Type":"ContainerStarted","Data":"982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0"} Oct 30 05:59:23.679306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.679277 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c248106-d4bc-4ede-ba9f-552da4945b3c" path="/var/lib/kubelet/pods/5c248106-d4bc-4ede-ba9f-552da4945b3c/volumes" Oct 30 05:59:23.679667 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.679649 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 05:59:23.705461 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:23.705421 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" podStartSLOduration=2.705408525 podStartE2EDuration="2.705408525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:23.702037199 +0000 UTC m=+1622.142414393" watchObservedRunningTime="2026-10-30 05:59:23.705408525 +0000 UTC m=+1622.145785717" Oct 30 05:59:24.675841 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:24.675790 6171 generic.go:356] "Generic (PLEG): container finished" podID="cfc26d67-28d8-41ee-a77f-11d2ab469596" containerID="3b99719f3da86f9fd2eba03443dfdde47b85b5bf73dfda28f6052d7df7f8306d" exitCode=0 Oct 30 05:59:24.676848 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:24.676815 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" event={"ID":"cfc26d67-28d8-41ee-a77f-11d2ab469596","Type":"ContainerDied","Data":"3b99719f3da86f9fd2eba03443dfdde47b85b5bf73dfda28f6052d7df7f8306d"} Oct 30 05:59:25.869124 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.869101 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:25.969294 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.969268 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\") pod \"cfc26d67-28d8-41ee-a77f-11d2ab469596\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " Oct 30 05:59:25.969434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.969327 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\" (UniqueName: \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\") pod \"cfc26d67-28d8-41ee-a77f-11d2ab469596\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " Oct 30 05:59:25.969512 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.969465 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\") pod \"cfc26d67-28d8-41ee-a77f-11d2ab469596\" (UID: \"cfc26d67-28d8-41ee-a77f-11d2ab469596\") " Oct 30 05:59:25.969673 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.969649 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca" pod "cfc26d67-28d8-41ee-a77f-11d2ab469596" (UID: "cfc26d67-28d8-41ee-a77f-11d2ab469596"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:25.970162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.970143 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:25.971731 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.971706 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg" pod "cfc26d67-28d8-41ee-a77f-11d2ab469596" (UID: "cfc26d67-28d8-41ee-a77f-11d2ab469596"). InnerVolumeSpecName "kube-api-access-tdwvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:25.971815 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:25.971740 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca" pod "cfc26d67-28d8-41ee-a77f-11d2ab469596" (UID: "cfc26d67-28d8-41ee-a77f-11d2ab469596"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:26.071835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:26.071806 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cfc26d67-28d8-41ee-a77f-11d2ab469596-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:26.071835 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:26.071833 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdwvg\" (UniqueName: \"kubernetes.io/projected/cfc26d67-28d8-41ee-a77f-11d2ab469596-kube-api-access-tdwvg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:26.395613 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:26.395569 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:59:26.695971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:26.695887 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" Oct 30 05:59:26.695971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:26.695922 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp" event={"ID":"cfc26d67-28d8-41ee-a77f-11d2ab469596","Type":"ContainerDied","Data":"982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0"} Oct 30 05:59:26.695971 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:26.695967 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982c6195e83d583e8932e31a396123d331f540f4d30191be15ac7ad1921a17d0" Oct 30 05:59:31.177660 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.177624 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 05:59:31.178465 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.178451 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cfc26d67-28d8-41ee-a77f-11d2ab469596" containerName="revoke-keys" Oct 30 05:59:31.178521 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.178467 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc26d67-28d8-41ee-a77f-11d2ab469596" containerName="revoke-keys" Oct 30 05:59:31.178635 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.178626 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc26d67-28d8-41ee-a77f-11d2ab469596" containerName="revoke-keys" Oct 30 05:59:31.184354 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.184335 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.207800 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.207777 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-84dc9d-serving-cert\"" Oct 30 05:59:31.222329 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.222302 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 05:59:31.250818 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.250783 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.250962 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.250913 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.251170 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.251150 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.251240 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.251225 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.352638 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.352605 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.352813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.352755 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.352813 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.352791 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.352887 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.352865 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.353509 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.353482 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.355360 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.355332 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.355516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.355434 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.378185 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.378131 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\") pod \"maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.501688 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.501351 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:31.868014 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:31.867988 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb0da49d_891d_4c77_baf6_e4596cd2071d.slice/crio-aa070dd2dbaa492e74478e16b889b8f6067997f5727c8c54140375402358fc13 WatchSource:0}: Error finding container aa070dd2dbaa492e74478e16b889b8f6067997f5727c8c54140375402358fc13: Status 404 returned error can't find the container with id aa070dd2dbaa492e74478e16b889b8f6067997f5727c8c54140375402358fc13 Oct 30 05:59:31.868955 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.868932 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 05:59:31.869466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:31.869451 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:59:32.806199 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:32.806160 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" event={"ID":"eb0da49d-891d-4c77-baf6-e4596cd2071d","Type":"ContainerStarted","Data":"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8"} Oct 30 05:59:32.806608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:32.806214 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" event={"ID":"eb0da49d-891d-4c77-baf6-e4596cd2071d","Type":"ContainerStarted","Data":"aa070dd2dbaa492e74478e16b889b8f6067997f5727c8c54140375402358fc13"} Oct 30 05:59:32.806608 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:32.806432 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:32.858433 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:32.858381 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" podStartSLOduration=1.560547373 podStartE2EDuration="1.858365657s" totalImagesPullingTime="297.818284ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:32.853807013 +0000 UTC m=+1631.294184207" watchObservedRunningTime="2026-10-30 05:59:32.858365657 +0000 UTC m=+1631.298742851" Oct 30 05:59:38.820719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:38.820689 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 05:59:50.012302 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.012271 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 05:59:50.017640 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.017618 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.043544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.043518 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 05:59:50.052162 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.052123 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-f4e35c-openshift-default-dockercfg-dvbfx\"" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172100 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172185 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172341 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172383 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172479 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172582 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172666 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172701 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.173113 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.172757 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtx79\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.277653 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.277560 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.278503 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.278478 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.279027 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.279010 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.279893 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.279875 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.280520 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.280497 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.281095 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.281073 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.281711 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.281692 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtx79\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.282262 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.282243 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.305977 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.305940 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.306838 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.279748 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.323012 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.278410 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.323335 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.280948 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.323449 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.282667 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.323558 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.278886 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.355075 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.339573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.373078 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.372818 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtx79\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.373308 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.373096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.412067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.411888 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\") pod \"e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.638758 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.638689 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:50.987920 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:50.987892 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfde1455_eedf_4b3c_bca0_e8a9ea76df05.slice/crio-d1200f9427b869fe4fe568dbb3db69696380a8020a8374f618ec6a191e7e085b WatchSource:0}: Error finding container d1200f9427b869fe4fe568dbb3db69696380a8020a8374f618ec6a191e7e085b: Status 404 returned error can't find the container with id d1200f9427b869fe4fe568dbb3db69696380a8020a8374f618ec6a191e7e085b Oct 30 05:59:50.988511 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.988364 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 05:59:50.990665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.990627 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:50.990789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.990718 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:50.990789 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:50.990762 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 05:59:51.117606 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:51.117562 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerStarted","Data":"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c"} Oct 30 05:59:51.118003 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:51.117687 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerStarted","Data":"d1200f9427b869fe4fe568dbb3db69696380a8020a8374f618ec6a191e7e085b"} Oct 30 05:59:51.639713 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:51.639679 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:52.427431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.427362 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podStartSLOduration=3.427342119 podStartE2EDuration="3.427342119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:51.171434097 +0000 UTC m=+1649.611811290" watchObservedRunningTime="2026-10-30 05:59:52.427342119 +0000 UTC m=+1650.867719314" Oct 30 05:59:52.435230 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.435184 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:59:52.435712 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.435676 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" containerID="cri-o://c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc" gracePeriod=30 Oct 30 05:59:52.441422 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.441399 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:59:52.441604 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.441572 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" containerID="cri-o://4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239" gracePeriod=30 Oct 30 05:59:52.643120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.636738 6171 patch_prober.go:29] interesting pod/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:59:52.643120 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.636803 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:59:52.708382 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:52.708305 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:53.048857 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.048817 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:59:53.049119 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.049064 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" podUID="3d35d538-9e00-40d2-aecb-8ed882773fed" containerName="maas-api" containerID="cri-o://1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f" gracePeriod=30 Oct 30 05:59:53.135289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.135259 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:53.136301 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.136283 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 05:59:53.577089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.575248 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:59:53.750129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.750095 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\") pod \"3d35d538-9e00-40d2-aecb-8ed882773fed\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " Oct 30 05:59:53.750493 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.750468 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\") pod \"3d35d538-9e00-40d2-aecb-8ed882773fed\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " Oct 30 05:59:53.777659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.763634 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\") pod \"3d35d538-9e00-40d2-aecb-8ed882773fed\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " Oct 30 05:59:53.777659 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.763705 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\" (UniqueName: \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\") pod \"3d35d538-9e00-40d2-aecb-8ed882773fed\" (UID: \"3d35d538-9e00-40d2-aecb-8ed882773fed\") " Oct 30 05:59:53.790829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.790162 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls" pod "3d35d538-9e00-40d2-aecb-8ed882773fed" (UID: "3d35d538-9e00-40d2-aecb-8ed882773fed"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:53.790829 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.790352 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j" pod "3d35d538-9e00-40d2-aecb-8ed882773fed" (UID: "3d35d538-9e00-40d2-aecb-8ed882773fed"). InnerVolumeSpecName "kube-api-access-cv22j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:53.795312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.791373 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca" pod "3d35d538-9e00-40d2-aecb-8ed882773fed" (UID: "3d35d538-9e00-40d2-aecb-8ed882773fed"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:53.795312 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.793629 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs" pod "3d35d538-9e00-40d2-aecb-8ed882773fed" (UID: "3d35d538-9e00-40d2-aecb-8ed882773fed"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:53.869957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.869926 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:53.869957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.869955 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d35d538-9e00-40d2-aecb-8ed882773fed-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:53.869957 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.869964 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv22j\" (UniqueName: \"kubernetes.io/projected/3d35d538-9e00-40d2-aecb-8ed882773fed-kube-api-access-cv22j\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:53.870315 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:53.869973 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d35d538-9e00-40d2-aecb-8ed882773fed-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:54.035979 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.035890 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds"] Oct 30 05:59:54.040092 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.040062 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3d35d538-9e00-40d2-aecb-8ed882773fed" containerName="maas-api" Oct 30 05:59:54.052287 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.052251 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d35d538-9e00-40d2-aecb-8ed882773fed" containerName="maas-api" Oct 30 05:59:54.055969 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.055937 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d35d538-9e00-40d2-aecb-8ed882773fed" containerName="maas-api" Oct 30 05:59:54.097309 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.097273 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds"] Oct 30 05:59:54.098657 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.097667 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.154821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.154570 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:59:54.154821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.154819 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 05:59:54.167218 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.167191 6171 generic.go:356] "Generic (PLEG): container finished" podID="3d35d538-9e00-40d2-aecb-8ed882773fed" containerID="1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f" exitCode=0 Oct 30 05:59:54.167400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.167254 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" Oct 30 05:59:54.167400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.167283 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" event={"ID":"3d35d538-9e00-40d2-aecb-8ed882773fed","Type":"ContainerDied","Data":"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f"} Oct 30 05:59:54.167400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.167330 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk" event={"ID":"3d35d538-9e00-40d2-aecb-8ed882773fed","Type":"ContainerDied","Data":"e171275ead755414914d244c96de276e7d7022e137922cf90e08f0b538ee550a"} Oct 30 05:59:54.167400 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.167355 6171 scope.go:122] "RemoveContainer" containerID="1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f" Oct 30 05:59:54.189286 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.189260 6171 scope.go:122] "RemoveContainer" containerID="1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f" Oct 30 05:59:54.189968 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:54.189930 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f\": container with ID starting with 1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f not found: ID does not exist" containerID="1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f" Oct 30 05:59:54.190192 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.190163 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f"} err="failed to get container status \"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f\": rpc error: code = NotFound desc = could not find container \"1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f\": container with ID starting with 1ed17c260fa63a1b4aa1e9b8aa4739fd3fafbc94db5254820a497578b516733f not found: ID does not exist" Oct 30 05:59:54.190320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.190242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.190447 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.190397 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.190687 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.190671 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.234768 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.234739 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:59:54.242769 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.242745 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-b202a0-77fdbcd45b-mpfxk"] Oct 30 05:59:54.293317 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.292991 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.293477 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.293426 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.293544 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.293510 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.294284 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.294217 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.296325 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.296296 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.322015 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.321990 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\") pod \"maas-api-revoke-sub-ae0f0b17-pntds\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.465071 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.464915 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:54.785084 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:54.785013 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bef560e_5b00_4911_a2ce_460b83332f24.slice/crio-02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d WatchSource:0}: Error finding container 02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d: Status 404 returned error can't find the container with id 02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d Oct 30 05:59:54.785865 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:54.785840 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds"] Oct 30 05:59:55.180035 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:55.180007 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" event={"ID":"1bef560e-5b00-4911-a2ce-460b83332f24","Type":"ContainerStarted","Data":"553a45841bc86163b107cbfa3ade2215958740ec3c161be7bf181060689744c4"} Oct 30 05:59:55.180244 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:55.180078 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" event={"ID":"1bef560e-5b00-4911-a2ce-460b83332f24","Type":"ContainerStarted","Data":"02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d"} Oct 30 05:59:55.228068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:55.228009 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" podStartSLOduration=2.227996268 podStartE2EDuration="2.227996268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:55.224658675 +0000 UTC m=+1653.665035869" watchObservedRunningTime="2026-10-30 05:59:55.227996268 +0000 UTC m=+1653.668373461" Oct 30 05:59:55.676438 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:55.676408 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d35d538-9e00-40d2-aecb-8ed882773fed" path="/var/lib/kubelet/pods/3d35d538-9e00-40d2-aecb-8ed882773fed/volumes" Oct 30 05:59:56.094697 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.094658 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 05:59:56.103466 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.103437 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.118363 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.118333 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 05:59:56.121681 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.121653 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdh8d\" (UniqueName: \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.121881 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.121863 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.135569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.135548 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f4e35c\"" Oct 30 05:59:56.135569 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.135552 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-f4e35c-dockercfg-5fp6v\"" Oct 30 05:59:56.158525 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.158502 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 05:59:56.163626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.163606 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.184216 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.184194 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 05:59:56.193320 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.193287 6171 generic.go:356] "Generic (PLEG): container finished" podID="1bef560e-5b00-4911-a2ce-460b83332f24" containerID="553a45841bc86163b107cbfa3ade2215958740ec3c161be7bf181060689744c4" exitCode=0 Oct 30 05:59:56.193434 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.193366 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" event={"ID":"1bef560e-5b00-4911-a2ce-460b83332f24","Type":"ContainerDied","Data":"553a45841bc86163b107cbfa3ade2215958740ec3c161be7bf181060689744c4"} Oct 30 05:59:56.223293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.223263 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.223460 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.223444 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.223614 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.223593 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdh8d\" (UniqueName: \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.223719 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.223706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjtjc\" (UniqueName: \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.224082 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.224065 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.310101 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.305467 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdh8d\" (UniqueName: \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\") pod \"payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.333068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.331628 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.333068 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.331842 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjtjc\" (UniqueName: \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.333371 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.333346 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.406129 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.380465 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjtjc\" (UniqueName: \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\") pod \"payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.426976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.426940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:56.480374 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.479427 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:56.625216 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:56.624441 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 05:59:56.948863 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:56.948835 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5e04b67_ef5c_4b28_b524_ddd5314edd85.slice/crio-69d9e08a6e4085db3a16eac0b5740ca65b441e3267197da0bfbae30a12e060bf WatchSource:0}: Error finding container 69d9e08a6e4085db3a16eac0b5740ca65b441e3267197da0bfbae30a12e060bf: Status 404 returned error can't find the container with id 69d9e08a6e4085db3a16eac0b5740ca65b441e3267197da0bfbae30a12e060bf Oct 30 05:59:56.950121 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.950097 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 05:59:56.961509 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:56.961487 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 05:59:56.965856 ip-10-0-136-35 kubenswrapper[6171]: W1030 05:59:56.965829 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e94acb_b40c_4c28_bede_9c24108a5d99.slice/crio-9efaaa214a1bde3c885ac22e3a5068ed7d417c201eac2ca23cf4adda63c36509 WatchSource:0}: Error finding container 9efaaa214a1bde3c885ac22e3a5068ed7d417c201eac2ca23cf4adda63c36509: Status 404 returned error can't find the container with id 9efaaa214a1bde3c885ac22e3a5068ed7d417c201eac2ca23cf4adda63c36509 Oct 30 05:59:57.204770 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.204671 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" event={"ID":"c5e04b67-ef5c-4b28-b524-ddd5314edd85","Type":"ContainerStarted","Data":"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef"} Oct 30 05:59:57.205238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.204782 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 05:59:57.205238 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.204806 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" event={"ID":"c5e04b67-ef5c-4b28-b524-ddd5314edd85","Type":"ContainerStarted","Data":"69d9e08a6e4085db3a16eac0b5740ca65b441e3267197da0bfbae30a12e060bf"} Oct 30 05:59:57.206514 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.206490 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" event={"ID":"87e94acb-b40c-4c28-bede-9c24108a5d99","Type":"ContainerStarted","Data":"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab"} Oct 30 05:59:57.206665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.206591 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 05:59:57.206665 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.206630 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" event={"ID":"87e94acb-b40c-4c28-bede-9c24108a5d99","Type":"ContainerStarted","Data":"9efaaa214a1bde3c885ac22e3a5068ed7d417c201eac2ca23cf4adda63c36509"} Oct 30 05:59:57.264067 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.264000 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" podStartSLOduration=1.263987252 podStartE2EDuration="1.263987252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:57.257272046 +0000 UTC m=+1655.697649240" watchObservedRunningTime="2026-10-30 05:59:57.263987252 +0000 UTC m=+1655.704364445" Oct 30 05:59:57.393237 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.393210 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:57.441384 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.441314 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" podStartSLOduration=1.441296625 podStartE2EDuration="1.441296625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:59:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:59:57.323102631 +0000 UTC m=+1655.763479824" watchObservedRunningTime="2026-10-30 05:59:57.441296625 +0000 UTC m=+1655.881673819" Oct 30 05:59:57.502169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.501281 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\") pod \"1bef560e-5b00-4911-a2ce-460b83332f24\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " Oct 30 05:59:57.502169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.501499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\") pod \"1bef560e-5b00-4911-a2ce-460b83332f24\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " Oct 30 05:59:57.502169 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.501626 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\") pod \"1bef560e-5b00-4911-a2ce-460b83332f24\" (UID: \"1bef560e-5b00-4911-a2ce-460b83332f24\") " Oct 30 05:59:57.506089 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.504111 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca" pod "1bef560e-5b00-4911-a2ce-460b83332f24" (UID: "1bef560e-5b00-4911-a2ce-460b83332f24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:57.565097 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.549531 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n" pod "1bef560e-5b00-4911-a2ce-460b83332f24" (UID: "1bef560e-5b00-4911-a2ce-460b83332f24"). InnerVolumeSpecName "kube-api-access-qqg7n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:57.565097 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.549621 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca" pod "1bef560e-5b00-4911-a2ce-460b83332f24" (UID: "1bef560e-5b00-4911-a2ce-460b83332f24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:57.605342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.604764 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqg7n\" (UniqueName: \"kubernetes.io/projected/1bef560e-5b00-4911-a2ce-460b83332f24-kube-api-access-qqg7n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.605342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.604797 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.605342 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.604812 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1bef560e-5b00-4911-a2ce-460b83332f24-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.734836 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.734773 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:59:57.765639 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.765618 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:59:57.807289 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807260 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807314 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807385 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807354 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807386 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807409 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807433 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807526 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.807663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807600 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.807663 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807638 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807689 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807739 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\") pod \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\" (UID: \"de0584cf-4f5a-4280-ae92-22c2cdf5e060\") " Oct 30 05:59:57.807832 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807742 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.807976 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807823 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:57.808029 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.807982 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.808508 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.808489 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.808647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.808512 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.808647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.808524 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.808647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.808536 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.808647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.808549 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.810010 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.809990 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:57.810150 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.810019 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:57.810290 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.810271 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.810431 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.810416 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst" pod "de0584cf-4f5a-4280-ae92-22c2cdf5e060" (UID: "de0584cf-4f5a-4280-ae92-22c2cdf5e060"). InnerVolumeSpecName "kube-api-access-6rnst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:57.909963 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.909920 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910139 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.909994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910140 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910205 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910170 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910264 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910321 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910308 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910443 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910352 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.910502 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910477 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910572 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910554 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910683 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910660 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\") pod \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\" (UID: \"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32\") " Oct 30 05:59:57.910755 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.910679 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.911516 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911472 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911576 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911598 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911613 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911626 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rnst\" (UniqueName: \"kubernetes.io/projected/de0584cf-4f5a-4280-ae92-22c2cdf5e060-kube-api-access-6rnst\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911639 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.911661 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911652 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de0584cf-4f5a-4280-ae92-22c2cdf5e060-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:57.912172 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.911849 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.912495 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.912462 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:57.912928 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.912904 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:57.913015 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.912980 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "kube-api-access-jjgp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:57.913271 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.913252 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:57.914272 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:57.914253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo" pod "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" (UID: "ace697a1-bb27-4c18-a1e7-3e3ba57dfd32"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014071 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjgp7\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-kube-api-access-jjgp7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014117 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014131 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014143 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014157 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014171 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.014264 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.014185 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 05:59:58.216766 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.216735 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" Oct 30 05:59:58.217193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.216763 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds" event={"ID":"1bef560e-5b00-4911-a2ce-460b83332f24","Type":"ContainerDied","Data":"02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d"} Oct 30 05:59:58.217193 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.216819 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02b3e090dd41e880ec874b3919267e694d6ebd7d90114a6285f093e19223e76d" Oct 30 05:59:58.219096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.218965 6171 generic.go:356] "Generic (PLEG): container finished" podID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerID="c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc" exitCode=0 Oct 30 05:59:58.219096 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.219073 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" Oct 30 05:59:58.219293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.219102 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerDied","Data":"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc"} Oct 30 05:59:58.219293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.219152 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm" event={"ID":"ace697a1-bb27-4c18-a1e7-3e3ba57dfd32","Type":"ContainerDied","Data":"2dd180ce77031886a307ffb8426c9822d1f14371a2d6f337ea862e82855f772b"} Oct 30 05:59:58.219293 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.219184 6171 scope.go:122] "RemoveContainer" containerID="c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc" Oct 30 05:59:58.220844 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.220818 6171 generic.go:356] "Generic (PLEG): container finished" podID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerID="4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239" exitCode=0 Oct 30 05:59:58.220972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.220865 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" Oct 30 05:59:58.220972 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.220903 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" event={"ID":"de0584cf-4f5a-4280-ae92-22c2cdf5e060","Type":"ContainerDied","Data":"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239"} Oct 30 05:59:58.221147 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.221037 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp" event={"ID":"de0584cf-4f5a-4280-ae92-22c2cdf5e060","Type":"ContainerDied","Data":"5683473c37ee5968c969705d9c9a4ea75cd5ca6315cd22a27cda4ab0a3c3550a"} Oct 30 05:59:58.232081 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.231886 6171 scope.go:122] "RemoveContainer" containerID="11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f" Oct 30 05:59:58.242821 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.242806 6171 scope.go:122] "RemoveContainer" containerID="c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc" Oct 30 05:59:58.243101 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:58.243085 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc\": container with ID starting with c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc not found: ID does not exist" containerID="c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc" Oct 30 05:59:58.243160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.243108 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc"} err="failed to get container status \"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc\": rpc error: code = NotFound desc = could not find container \"c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc\": container with ID starting with c566c56cf0ef3da2e7d49687e594e74ef984dc6035c91ec22213e3abd00878dc not found: ID does not exist" Oct 30 05:59:58.243160 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.243123 6171 scope.go:122] "RemoveContainer" containerID="11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f" Oct 30 05:59:58.243348 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:58.243334 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f\": container with ID starting with 11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f not found: ID does not exist" containerID="11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f" Oct 30 05:59:58.243386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.243351 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f"} err="failed to get container status \"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f\": rpc error: code = NotFound desc = could not find container \"11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f\": container with ID starting with 11e5818bf03a876be63eff90bf88baba8e85034ec8061f9b94ad89fbdad8be7f not found: ID does not exist" Oct 30 05:59:58.243386 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.243363 6171 scope.go:122] "RemoveContainer" containerID="4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239" Oct 30 05:59:58.253647 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.253631 6171 scope.go:122] "RemoveContainer" containerID="4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239" Oct 30 05:59:58.253945 ip-10-0-136-35 kubenswrapper[6171]: E1030 05:59:58.253924 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239\": container with ID starting with 4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239 not found: ID does not exist" containerID="4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239" Oct 30 05:59:58.254073 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.253949 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239"} err="failed to get container status \"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239\": rpc error: code = NotFound desc = could not find container \"4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239\": container with ID starting with 4fefddc681c381dbaa839be93b65d7ed672faf9b4f00638f4f7e971cf97c3239 not found: ID does not exist" Oct 30 05:59:58.283595 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.283542 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:59:58.293819 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.293803 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-5shmm"] Oct 30 05:59:58.346206 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.346176 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:59:58.352626 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:58.352028 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-1b2243ef-openshift-default-88dfb6485-psthp"] Oct 30 05:59:59.676306 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:59.676279 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" path="/var/lib/kubelet/pods/ace697a1-bb27-4c18-a1e7-3e3ba57dfd32/volumes" Oct 30 05:59:59.676765 ip-10-0-136-35 kubenswrapper[6171]: I1030 05:59:59.676750 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" path="/var/lib/kubelet/pods/de0584cf-4f5a-4280-ae92-22c2cdf5e060/volumes" Oct 30 06:00:00.152864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.152827 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:00.153746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153720 6171 cpu_manager.go:414] "containerMap: removing container" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" Oct 30 06:00:00.153868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153751 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" Oct 30 06:00:00.153868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153819 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1bef560e-5b00-4911-a2ce-460b83332f24" containerName="revoke-keys" Oct 30 06:00:00.153868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153830 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bef560e-5b00-4911-a2ce-460b83332f24" containerName="revoke-keys" Oct 30 06:00:00.153960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153894 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.153960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153902 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.153960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153921 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.153960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.153928 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.154182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.154169 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bef560e-5b00-4911-a2ce-460b83332f24" containerName="revoke-keys" Oct 30 06:00:00.154217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.154194 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.154217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.154203 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ace697a1-bb27-4c18-a1e7-3e3ba57dfd32" containerName="istio-proxy" Oct 30 06:00:00.154217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.154210 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="de0584cf-4f5a-4280-ae92-22c2cdf5e060" containerName="istio-proxy" Oct 30 06:00:00.159930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.159910 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.169459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.169434 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:00.175498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.175479 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:00.175745 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.175721 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.211985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.183822 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:00:00.211985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.185149 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.211985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.186513 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:00.211985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.198647 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:00.213735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.213453 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:00.223348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.214107 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.229096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.226806 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:00.229096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.226844 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:00.229096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.226923 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.229096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.227125 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:00.288513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.288479 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.288759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.288737 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8lgs\" (UniqueName: \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.288927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.288907 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-57jh7\" (UniqueName: \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.289333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.289310 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.293259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.293236 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:00.293374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.293342 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.304732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.304098 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:00.313895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.313817 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:00.321453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.321409 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:00.328033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.328000 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.329515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.329491 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:00.390560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390527 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.390767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390567 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8lgs\" (UniqueName: \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.390767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390747 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.390898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390796 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6vm\" (UniqueName: \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.390967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390891 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-57jh7\" (UniqueName: \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.391027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.390962 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx86g\" (UniqueName: \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.391140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391082 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.391203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391142 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dfrz\" (UniqueName: \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.391261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391244 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qvz\" (UniqueName: \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.391335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391315 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.391397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391328 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.391483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391466 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.391548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391511 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.391548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391541 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5wxx\" (UniqueName: \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.391663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.391560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.392183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.392161 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.422569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.422489 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8lgs\" (UniqueName: \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\") pod \"maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.423717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.423625 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-57jh7\" (UniqueName: \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\") pod \"maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.487399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.487369 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:00.492495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492467 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.492623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492506 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5wxx\" (UniqueName: \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.492623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492525 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.492751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492682 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.492751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492716 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6vm\" (UniqueName: \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.492884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492822 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx86g\" (UniqueName: \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.492941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492913 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.492999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.492969 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dfrz\" (UniqueName: \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.493279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493003 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qvz\" (UniqueName: \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.493279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493110 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.493500 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493476 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.493666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493646 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.493968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493949 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.494029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.493955 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.494183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.494164 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.514952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.514924 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:00.519107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.519080 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dfrz\" (UniqueName: \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.519226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.519082 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6vm\" (UniqueName: \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\") pod \"maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.520083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.520032 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx86g\" (UniqueName: \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\") pod \"maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.522180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.522151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qvz\" (UniqueName: \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.525174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.525152 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5wxx\" (UniqueName: \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.595379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.595345 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:00.598892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.598642 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:00.610207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.609891 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:00.618144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.617807 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:00.650286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.648186 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:00.826232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.823985 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:00.885228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:00.885197 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:01.208640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.208612 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 06:00:01.209298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.209248 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerName="maas-api" containerID="cri-o://2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536" gracePeriod=30 Oct 30 06:00:01.227863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.227838 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:01.250081 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:01.248085 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8814ac67_80a5_44d0_b25c_5751cc589ccf.slice/crio-1343a432c9fae7b054568ef1e2006cd1b67db3dc469a014c39579fd61b58308e WatchSource:0}: Error finding container 1343a432c9fae7b054568ef1e2006cd1b67db3dc469a014c39579fd61b58308e: Status 404 returned error can't find the container with id 1343a432c9fae7b054568ef1e2006cd1b67db3dc469a014c39579fd61b58308e Oct 30 06:00:01.260871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.260845 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:01.271570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.271542 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:01.272916 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:01.272882 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c5ed71_35d2_441a_a2b3_f3332c7c0980.slice/crio-df79760194a4247444132333c079a68edda3697a5dd21c74652b6ca7c2a3b4fb WatchSource:0}: Error finding container df79760194a4247444132333c079a68edda3697a5dd21c74652b6ca7c2a3b4fb: Status 404 returned error can't find the container with id df79760194a4247444132333c079a68edda3697a5dd21c74652b6ca7c2a3b4fb Oct 30 06:00:01.277160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.277138 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:01.292611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.292584 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:01.305584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.305556 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerStarted","Data":"7e46815afac22ab852412b2712f87f7e3787805b58203ab8e7c819ee84b92bc5"} Oct 30 06:00:01.307088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.307067 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerStarted","Data":"fef07cbe3d3f6ef24cbbe19bb20049acd6c16d25ac892acada3a3262ab1ef32e"} Oct 30 06:00:01.384207 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:01.384179 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64a1b9f6_c4d4_433e_83d0_742c0a406ed4.slice/crio-1443fd454848eeab03651d114eb731b3fe053a93afc6d51c41821567f3515af1 WatchSource:0}: Error finding container 1443fd454848eeab03651d114eb731b3fe053a93afc6d51c41821567f3515af1: Status 404 returned error can't find the container with id 1443fd454848eeab03651d114eb731b3fe053a93afc6d51c41821567f3515af1 Oct 30 06:00:01.584270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.584237 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 06:00:01.635577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.619509 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\") pod \"371c1e7c-c543-44df-b781-08b79e15ecb7\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " Oct 30 06:00:01.635577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.619579 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\") pod \"371c1e7c-c543-44df-b781-08b79e15ecb7\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " Oct 30 06:00:01.635577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.619658 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\") pod \"371c1e7c-c543-44df-b781-08b79e15ecb7\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " Oct 30 06:00:01.635577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.619715 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\" (UniqueName: \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\") pod \"371c1e7c-c543-44df-b781-08b79e15ecb7\" (UID: \"371c1e7c-c543-44df-b781-08b79e15ecb7\") " Oct 30 06:00:01.652378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.636143 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca" pod "371c1e7c-c543-44df-b781-08b79e15ecb7" (UID: "371c1e7c-c543-44df-b781-08b79e15ecb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:01.732092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.730228 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/371c1e7c-c543-44df-b781-08b79e15ecb7-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:01.819012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.817236 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj" pod "371c1e7c-c543-44df-b781-08b79e15ecb7" (UID: "371c1e7c-c543-44df-b781-08b79e15ecb7"). InnerVolumeSpecName "kube-api-access-gftpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:01.834153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.833001 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gftpj\" (UniqueName: \"kubernetes.io/projected/371c1e7c-c543-44df-b781-08b79e15ecb7-kube-api-access-gftpj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:01.841761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.840203 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs" pod "371c1e7c-c543-44df-b781-08b79e15ecb7" (UID: "371c1e7c-c543-44df-b781-08b79e15ecb7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:00:01.841761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.840226 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls" pod "371c1e7c-c543-44df-b781-08b79e15ecb7" (UID: "371c1e7c-c543-44df-b781-08b79e15ecb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:00:01.902628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.902589 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:00:01.969844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.969689 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:01.969844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:01.969721 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/371c1e7c-c543-44df-b781-08b79e15ecb7-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:02.317905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.317874 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerStarted","Data":"b58d56de8fa75cf73e01d0e11b52ff662e3bba51ee0fc6e4c4e25fc673627105"} Oct 30 06:00:02.318104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.317936 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerStarted","Data":"627b6bb4a48eca39f80c81fde8622bb1363819e45f4edc199c08596aad975d87"} Oct 30 06:00:02.319653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.319621 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerStarted","Data":"4716240d7558ba45e15aeacfd6bcac5edcb6b0d84fb83a5bfa2299126bb3ae38"} Oct 30 06:00:02.319766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.319676 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerStarted","Data":"df79760194a4247444132333c079a68edda3697a5dd21c74652b6ca7c2a3b4fb"} Oct 30 06:00:02.321288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.321259 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerStarted","Data":"4360d883b542024b4a1e873aae831e301ab5797ed80a6c3f93317b03fc55ce62"} Oct 30 06:00:02.321389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.321313 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerStarted","Data":"1443fd454848eeab03651d114eb731b3fe053a93afc6d51c41821567f3515af1"} Oct 30 06:00:02.322894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.322873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerStarted","Data":"a61f4b346209ddfa8c501fac61aa18cf2a183313999a98a2954bb877dd960126"} Oct 30 06:00:02.322997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.322908 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerStarted","Data":"1343a432c9fae7b054568ef1e2006cd1b67db3dc469a014c39579fd61b58308e"} Oct 30 06:00:02.324518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.324489 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerStarted","Data":"a8a2450085f8d80877b7b3954c716c989661aceeb447c73f3e7407f2ab761d27"} Oct 30 06:00:02.325082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.324535 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerStarted","Data":"2c3989e7e43b2d2a9b24abdfa49c879fc4b69f05460fdf5baae1848fe0396260"} Oct 30 06:00:02.327092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.327067 6171 generic.go:356] "Generic (PLEG): container finished" podID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerID="2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536" exitCode=0 Oct 30 06:00:02.327227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.327214 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" Oct 30 06:00:02.327873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.327853 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" event={"ID":"371c1e7c-c543-44df-b781-08b79e15ecb7","Type":"ContainerDied","Data":"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536"} Oct 30 06:00:02.327950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.327896 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" event={"ID":"371c1e7c-c543-44df-b781-08b79e15ecb7","Type":"ContainerDied","Data":"384165c8ece3af4e13f66ac1f689f0a954c59268061c7d8fac31f527ae7d3c54"} Oct 30 06:00:02.327950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.327923 6171 scope.go:122] "RemoveContainer" containerID="2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536" Oct 30 06:00:02.332021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.331997 6171 generic.go:356] "Generic (PLEG): container finished" podID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerID="b102d6069151c968deb3def9f61a63dd23d29a85bade4481222a3384044b5d52" exitCode=60 Oct 30 06:00:02.332136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.332117 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerDied","Data":"b102d6069151c968deb3def9f61a63dd23d29a85bade4481222a3384044b5d52"} Oct 30 06:00:02.332350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.332333 6171 scope.go:122] "RemoveContainer" containerID="b102d6069151c968deb3def9f61a63dd23d29a85bade4481222a3384044b5d52" Oct 30 06:00:02.334395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.334262 6171 generic.go:356] "Generic (PLEG): container finished" podID="0becf319-257f-4b48-87d6-788e15435d0e" containerID="4b61bd7af11ec535f9bf610619dc72c92293f2e558dd28ef0e48c832e93c44cd" exitCode=60 Oct 30 06:00:02.334395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.334301 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerDied","Data":"4b61bd7af11ec535f9bf610619dc72c92293f2e558dd28ef0e48c832e93c44cd"} Oct 30 06:00:02.334621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.334604 6171 scope.go:122] "RemoveContainer" containerID="4b61bd7af11ec535f9bf610619dc72c92293f2e558dd28ef0e48c832e93c44cd" Oct 30 06:00:02.354795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.354776 6171 scope.go:122] "RemoveContainer" containerID="2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536" Oct 30 06:00:02.355196 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:02.355124 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536\": container with ID starting with 2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536 not found: ID does not exist" containerID="2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536" Oct 30 06:00:02.355196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.355149 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536"} err="failed to get container status \"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536\": rpc error: code = NotFound desc = could not find container \"2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536\": container with ID starting with 2cf7d5c563e34f8d7208af6c1a36288db86d910a77c735c82175b22c5d1cf536 not found: ID does not exist" Oct 30 06:00:02.376600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.376545 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" podStartSLOduration=2.3765289960000002 podStartE2EDuration="2.376528996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.371091677 +0000 UTC m=+1660.811468872" watchObservedRunningTime="2026-10-30 06:00:02.376528996 +0000 UTC m=+1660.816906186" Oct 30 06:00:02.425292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.425238 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" podStartSLOduration=2.425221419 podStartE2EDuration="2.425221419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.423940478 +0000 UTC m=+1660.864317671" watchObservedRunningTime="2026-10-30 06:00:02.425221419 +0000 UTC m=+1660.865598615" Oct 30 06:00:02.477413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.477350 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" podStartSLOduration=2.477333344 podStartE2EDuration="2.477333344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.475451352 +0000 UTC m=+1660.915828545" watchObservedRunningTime="2026-10-30 06:00:02.477333344 +0000 UTC m=+1660.917710537" Oct 30 06:00:02.663375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.663343 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 06:00:02.673196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.671395 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj"] Oct 30 06:00:02.700505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.700454 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" podStartSLOduration=2.700440147 podStartE2EDuration="2.700440147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.692902874 +0000 UTC m=+1661.133280067" watchObservedRunningTime="2026-10-30 06:00:02.700440147 +0000 UTC m=+1661.140817339" Oct 30 06:00:02.739981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:02.739926 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" podStartSLOduration=2.739912211 podStartE2EDuration="2.739912211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.736418569 +0000 UTC m=+1661.176795763" watchObservedRunningTime="2026-10-30 06:00:02.739912211 +0000 UTC m=+1661.180289404" Oct 30 06:00:03.424132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.420512 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerStarted","Data":"9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a"} Oct 30 06:00:03.427076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.424944 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerStarted","Data":"091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93"} Oct 30 06:00:03.435063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.431836 6171 generic.go:356] "Generic (PLEG): container finished" podID="9fac396a-8fbd-46d7-902b-662e1629b512" containerID="b58d56de8fa75cf73e01d0e11b52ff662e3bba51ee0fc6e4c4e25fc673627105" exitCode=60 Oct 30 06:00:03.435063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.431929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerDied","Data":"b58d56de8fa75cf73e01d0e11b52ff662e3bba51ee0fc6e4c4e25fc673627105"} Oct 30 06:00:03.435063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.432426 6171 scope.go:122] "RemoveContainer" containerID="b58d56de8fa75cf73e01d0e11b52ff662e3bba51ee0fc6e4c4e25fc673627105" Oct 30 06:00:03.445123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.445088 6171 generic.go:356] "Generic (PLEG): container finished" podID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerID="4716240d7558ba45e15aeacfd6bcac5edcb6b0d84fb83a5bfa2299126bb3ae38" exitCode=60 Oct 30 06:00:03.445271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.445209 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerDied","Data":"4716240d7558ba45e15aeacfd6bcac5edcb6b0d84fb83a5bfa2299126bb3ae38"} Oct 30 06:00:03.445748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.445719 6171 scope.go:122] "RemoveContainer" containerID="4716240d7558ba45e15aeacfd6bcac5edcb6b0d84fb83a5bfa2299126bb3ae38" Oct 30 06:00:03.449019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.448991 6171 generic.go:356] "Generic (PLEG): container finished" podID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerID="4360d883b542024b4a1e873aae831e301ab5797ed80a6c3f93317b03fc55ce62" exitCode=60 Oct 30 06:00:03.449283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.449261 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerDied","Data":"4360d883b542024b4a1e873aae831e301ab5797ed80a6c3f93317b03fc55ce62"} Oct 30 06:00:03.449922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.449892 6171 scope.go:122] "RemoveContainer" containerID="4360d883b542024b4a1e873aae831e301ab5797ed80a6c3f93317b03fc55ce62" Oct 30 06:00:03.455055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.454625 6171 generic.go:356] "Generic (PLEG): container finished" podID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerID="a61f4b346209ddfa8c501fac61aa18cf2a183313999a98a2954bb877dd960126" exitCode=60 Oct 30 06:00:03.455055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.454739 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerDied","Data":"a61f4b346209ddfa8c501fac61aa18cf2a183313999a98a2954bb877dd960126"} Oct 30 06:00:03.455563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.455215 6171 scope.go:122] "RemoveContainer" containerID="a61f4b346209ddfa8c501fac61aa18cf2a183313999a98a2954bb877dd960126" Oct 30 06:00:03.483682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.483618 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" podStartSLOduration=3.483596119 podStartE2EDuration="3.483596119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.47925355 +0000 UTC m=+1661.919630743" watchObservedRunningTime="2026-10-30 06:00:03.483596119 +0000 UTC m=+1661.923973315" Oct 30 06:00:03.493471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.493438 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerID="a8a2450085f8d80877b7b3954c716c989661aceeb447c73f3e7407f2ab761d27" exitCode=60 Oct 30 06:00:03.493697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.493678 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerDied","Data":"a8a2450085f8d80877b7b3954c716c989661aceeb447c73f3e7407f2ab761d27"} Oct 30 06:00:03.494386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.494305 6171 scope.go:122] "RemoveContainer" containerID="a8a2450085f8d80877b7b3954c716c989661aceeb447c73f3e7407f2ab761d27" Oct 30 06:00:03.562163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.562112 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" podStartSLOduration=3.562096672 podStartE2EDuration="3.562096672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.56020271 +0000 UTC m=+1662.000579904" watchObservedRunningTime="2026-10-30 06:00:03.562096672 +0000 UTC m=+1662.002473864" Oct 30 06:00:03.913185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:03.891002 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" path="/var/lib/kubelet/pods/371c1e7c-c543-44df-b781-08b79e15ecb7/volumes" Oct 30 06:00:04.507521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.507482 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4fc75f-5d5864445b-9ppgj" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.193:8443/health\": context deadline exceeded" Oct 30 06:00:04.517181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.517145 6171 generic.go:356] "Generic (PLEG): container finished" podID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerID="9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a" exitCode=60 Oct 30 06:00:04.517999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.517528 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerDied","Data":"9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a"} Oct 30 06:00:04.517999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.517614 6171 scope.go:122] "RemoveContainer" containerID="b102d6069151c968deb3def9f61a63dd23d29a85bade4481222a3384044b5d52" Oct 30 06:00:04.517999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.517784 6171 scope.go:122] "RemoveContainer" containerID="9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a" Oct 30 06:00:04.518307 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:04.518280 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch_odh-ai-gateway-infra(78d3f6b6-deef-4822-bac3-1adc7221f4f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" Oct 30 06:00:04.521733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.521711 6171 generic.go:356] "Generic (PLEG): container finished" podID="0becf319-257f-4b48-87d6-788e15435d0e" containerID="091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93" exitCode=60 Oct 30 06:00:04.521957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.521903 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerDied","Data":"091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93"} Oct 30 06:00:04.522373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.522165 6171 scope.go:122] "RemoveContainer" containerID="091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93" Oct 30 06:00:04.522504 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:04.522478 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t_odh-ai-gateway-infra(0becf319-257f-4b48-87d6-788e15435d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" podUID="0becf319-257f-4b48-87d6-788e15435d0e" Oct 30 06:00:04.535835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.535076 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerStarted","Data":"81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36"} Oct 30 06:00:04.542169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.542138 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerStarted","Data":"3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7"} Oct 30 06:00:04.542826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.542795 6171 scope.go:122] "RemoveContainer" containerID="4b61bd7af11ec535f9bf610619dc72c92293f2e558dd28ef0e48c832e93c44cd" Oct 30 06:00:04.549876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:04.549847 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerStarted","Data":"87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3"} Oct 30 06:00:05.594127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:05.589452 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:00:05.628554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:05.626908 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:05.644070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:05.641881 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:05.813085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:05.810586 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:05.884828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:05.881409 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.044177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.044131 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.136472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.136328 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.286934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.286893 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:00:06.319084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.289886 6171 cpu_manager.go:414] "containerMap: removing container" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerName="maas-api" Oct 30 06:00:06.319378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.319351 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerName="maas-api" Oct 30 06:00:06.320423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.320381 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="371c1e7c-c543-44df-b781-08b79e15ecb7" containerName="maas-api" Oct 30 06:00:06.500071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.483739 6171 scope.go:122] "RemoveContainer" containerID="9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a" Oct 30 06:00:06.500071 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:06.485570 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch_odh-ai-gateway-infra(78d3f6b6-deef-4822-bac3-1adc7221f4f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" Oct 30 06:00:06.500071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.495523 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.551655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.551593 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:00:06.552189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.552167 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.633305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.633261 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.644125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.644085 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:00:06.653081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.646172 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.653081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.646459 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:00:06.653081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.652421 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c1d09f-openshift-default-dockercfg-gpb22\"" Oct 30 06:00:06.676187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.654961 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:00:06.676187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.655033 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:00:06.676187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.655095 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:00:06.677553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.677521 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.677838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.677818 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.678093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.678072 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2gnc\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.678268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.678242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.678604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.678571 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.678812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.678778 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.679934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.679904 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.680129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.680094 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.706215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.706178 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.706790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.699731 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.742079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.717842 6171 scope.go:122] "RemoveContainer" containerID="091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93" Oct 30 06:00:06.742079 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:06.718322 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t_odh-ai-gateway-infra(0becf319-257f-4b48-87d6-788e15435d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" podUID="0becf319-257f-4b48-87d6-788e15435d0e" Oct 30 06:00:06.777286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.776685 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812291 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812351 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812421 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812593 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812680 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812763 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2gnc\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.812811 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.813066 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.815073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.813115 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.816318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.816178 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.846098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.816569 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.846098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.817110 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.846098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.818143 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.846098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.818653 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:06.886959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.886915 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.002292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:06.990275 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:00:07.086108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.085697 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.123080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.114398 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2gnc\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:07.138567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.123554 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:07.206738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.161642 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:07.284368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.170716 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:00:07.361157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.244764 6171 generic.go:356] "Generic (PLEG): container finished" podID="9fac396a-8fbd-46d7-902b-662e1629b512" containerID="53b0a85d2b28b034ad349a74d0b96f4493db54028e8dd228dbed9c2e6b72e527" exitCode=60 Oct 30 06:00:07.420075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.327609 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:07.430579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.430549 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:07.484163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.484110 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.510432 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.510389 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.658013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.639228 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:00:07.658013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.647127 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.658013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.647624 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:00:07.658013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.647985 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.701207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.701167 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:00:07.701465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.701437 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.736085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.727381 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.773814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.773782 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.798078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.779980 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.803547 6171 generic.go:356] "Generic (PLEG): container finished" podID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerID="81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36" exitCode=60 Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806004 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806166 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806215 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806246 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806273 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806337 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806363 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnfd\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806394 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.806621 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.825085 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.825180 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerDied","Data":"53b0a85d2b28b034ad349a74d0b96f4493db54028e8dd228dbed9c2e6b72e527"} Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.825224 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerDied","Data":"81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36"} Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.825369 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.831417 6171 scope.go:122] "RemoveContainer" containerID="b58d56de8fa75cf73e01d0e11b52ff662e3bba51ee0fc6e4c4e25fc673627105" Oct 30 06:00:07.849078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.844859 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-aee070-openshift-default-dockercfg-stq9x\"" Oct 30 06:00:07.876488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.851840 6171 scope.go:122] "RemoveContainer" containerID="81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36" Oct 30 06:00:07.876488 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:07.868306 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k_odh-ai-gateway-infra(91c5ed71-35d2-441a-a2b3-f3332c7c0980)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" Oct 30 06:00:07.876488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.868369 6171 scope.go:122] "RemoveContainer" containerID="53b0a85d2b28b034ad349a74d0b96f4493db54028e8dd228dbed9c2e6b72e527" Oct 30 06:00:07.876488 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:07.868680 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz_odh-ai-gateway-infra(9fac396a-8fbd-46d7-902b-662e1629b512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.929735 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.929857 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.929903 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.929938 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.930029 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.930080 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnfd\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.947204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.930115 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.975073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.948515 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.976311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.976282 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:07.976527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.976509 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.014221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.993322 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.014221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.983204 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.014221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.993459 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.014221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.993961 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.014221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:07.994522 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.039153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.038398 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.070072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.050365 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.094190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.087753 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.253373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.237650 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.374087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.364820 6171 generic.go:356] "Generic (PLEG): container finished" podID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerID="99999aea9d937639b1929f201e3cdacd2b8aa40acafb64109554ba3fff8d36f2" exitCode=60 Oct 30 06:00:08.374087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.367943 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.374087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.368762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnfd\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\") pod \"e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.490069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.464831 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:00:08.490343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.490311 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerDied","Data":"99999aea9d937639b1929f201e3cdacd2b8aa40acafb64109554ba3fff8d36f2"} Oct 30 06:00:08.490577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.490557 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:00:08.490735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.490722 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:00:08.490882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.490870 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:00:08.491027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.491015 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 06:00:08.491215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.491201 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 06:00:08.493422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.493402 6171 scope.go:122] "RemoveContainer" containerID="99999aea9d937639b1929f201e3cdacd2b8aa40acafb64109554ba3fff8d36f2" Oct 30 06:00:08.505116 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:08.493822 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5_odh-ai-gateway-infra(64a1b9f6-c4d4-433e-83d0-742c0a406ed4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" Oct 30 06:00:08.507535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.507509 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:08.527381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.526693 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:00:08.635067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.629161 6171 generic.go:356] "Generic (PLEG): container finished" podID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerID="3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7" exitCode=60 Oct 30 06:00:08.635067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.629497 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerDied","Data":"3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7"} Oct 30 06:00:08.648956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.648652 6171 scope.go:122] "RemoveContainer" containerID="4716240d7558ba45e15aeacfd6bcac5edcb6b0d84fb83a5bfa2299126bb3ae38" Oct 30 06:00:08.650251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.650217 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:00:08.650764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.650735 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:00:08.694067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.682310 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:00:08.730070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.724084 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerID="87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3" exitCode=60 Oct 30 06:00:08.739776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.739702 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": context deadline exceeded" Oct 30 06:00:08.815163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.815035 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:08.822204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.822160 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerDied","Data":"87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3"} Oct 30 06:00:08.868199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.846507 6171 scope.go:122] "RemoveContainer" containerID="87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3" Oct 30 06:00:08.868539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:08.868520 6171 scope.go:122] "RemoveContainer" containerID="3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7" Oct 30 06:00:08.896793 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:08.896750 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276_odh-ai-gateway-infra(8814ac67-80a5-44d0-b25c-5751cc589ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" Oct 30 06:00:08.897022 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:08.870105 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm_odh-ai-gateway-infra(5f7922d7-702a-4a0a-a32a-fa8017c00540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" Oct 30 06:00:09.044070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.019297 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.259003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.099281 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.353935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.353903 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:00:09.431138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.431084 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.432225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.432190 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:09.493153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.477361 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:00:09.513284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.511150 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 06:00:09.517571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.517542 6171 scope.go:122] "RemoveContainer" containerID="4360d883b542024b4a1e873aae831e301ab5797ed80a6c3f93317b03fc55ce62" Oct 30 06:00:09.522999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.522978 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:00:09.523245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.523230 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:00:09.551093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.534355 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:00:09.551093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.546168 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:00:09.578976 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.575785 6171 scope.go:122] "RemoveContainer" containerID="a61f4b346209ddfa8c501fac61aa18cf2a183313999a98a2954bb877dd960126" Oct 30 06:00:09.687166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:09.661407 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:10.039816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.039685 6171 scope.go:122] "RemoveContainer" containerID="a8a2450085f8d80877b7b3954c716c989661aceeb447c73f3e7407f2ab761d27" Oct 30 06:00:10.055381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.053573 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.055381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.053745 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.055381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.053869 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.065059 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.065159 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.065230 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066530 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066609 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066644 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066681 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066715 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066743 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:00:10.067505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.066770 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:00:10.076903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.076880 6171 scope.go:122] "RemoveContainer" containerID="81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36" Oct 30 06:00:10.077430 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:10.077394 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k_odh-ai-gateway-infra(91c5ed71-35d2-441a-a2b3-f3332c7c0980)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" Oct 30 06:00:10.078928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.078811 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:00:10.118638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.118553 6171 scope.go:122] "RemoveContainer" containerID="87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3" Oct 30 06:00:10.119187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:10.118834 6171 scope.go:122] "RemoveContainer" containerID="3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7" Oct 30 06:00:10.119187 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:10.119089 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276_odh-ai-gateway-infra(8814ac67-80a5-44d0-b25c-5751cc589ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" Oct 30 06:00:10.119187 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:10.118828 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm_odh-ai-gateway-infra(5f7922d7-702a-4a0a-a32a-fa8017c00540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" Oct 30 06:00:11.136821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.136783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerStarted","Data":"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394"} Oct 30 06:00:11.137363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.136848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerStarted","Data":"9d7adef96824a639cd417d8e67e9af14b74bcaefe27fb1847e0c46666f6fe72e"} Oct 30 06:00:11.139103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.139077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerStarted","Data":"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657"} Oct 30 06:00:11.139259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.139123 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerStarted","Data":"d134c67db231a348b9654b59251faf658bab8d043741e071133fe313fa630e5e"} Oct 30 06:00:11.210859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.210797 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podStartSLOduration=7.210783468 podStartE2EDuration="7.210783468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:11.203342587 +0000 UTC m=+1669.643719781" watchObservedRunningTime="2026-10-30 06:00:11.210783468 +0000 UTC m=+1669.651160661" Oct 30 06:00:11.261797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.261765 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:00:11.263789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.263746 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podStartSLOduration=6.263731943 podStartE2EDuration="6.263731943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:11.256278955 +0000 UTC m=+1669.696656149" watchObservedRunningTime="2026-10-30 06:00:11.263731943 +0000 UTC m=+1669.704109135" Oct 30 06:00:11.268330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.268307 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.327856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.327815 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pd5w9\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.327874 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.327978 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328102 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328151 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328276 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328384 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328551 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.328657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.328606 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.333247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.333220 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:00:11.370021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.369994 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-f78ca5-openshift-default-dockercfg-2l8fk\"" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437330 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437393 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437572 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pd5w9\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437601 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437646 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437718 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437768 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437824 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.437888 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.438478 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.439271 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.444124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.439502 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.445271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.445244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.463072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.456657 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.469092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.467257 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:11.500655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.477192 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pd5w9\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.500999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.499390 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.501952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.501914 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.516218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.516191 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:11.562765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.546270 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\") pod \"e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.620080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.617396 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:11.634407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.634367 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:11.664541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:11.664511 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:12.160183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.159117 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:12.160183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.159176 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:12.160183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.159884 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:00:12.160183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.160191 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:00:12.366671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.366644 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:00:12.370563 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:12.370531 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5963eb_38cb_45a2_9ae8_7e8a60d398da.slice/crio-b3e2eda13380689d48aa75777bb9f04108263540658d6fec1bf7409805d531e3 WatchSource:0}: Error finding container b3e2eda13380689d48aa75777bb9f04108263540658d6fec1bf7409805d531e3: Status 404 returned error can't find the container with id b3e2eda13380689d48aa75777bb9f04108263540658d6fec1bf7409805d531e3 Oct 30 06:00:12.373652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.373612 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:12.373752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.373706 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:12.373813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.373757 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:12.442818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.442791 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 06:00:12.443093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.443018 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" containerID="cri-o://c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46" gracePeriod=30 Oct 30 06:00:12.452182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.452142 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 06:00:12.452332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:12.452310 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" containerID="cri-o://6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957" gracePeriod=30 Oct 30 06:00:13.183788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.183743 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerStarted","Data":"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9"} Oct 30 06:00:13.184298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.183849 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerStarted","Data":"b3e2eda13380689d48aa75777bb9f04108263540658d6fec1bf7409805d531e3"} Oct 30 06:00:13.239026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.238964 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podStartSLOduration=2.238946958 podStartE2EDuration="2.238946958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:13.236646934 +0000 UTC m=+1671.677024127" watchObservedRunningTime="2026-10-30 06:00:13.238946958 +0000 UTC m=+1671.679324154" Oct 30 06:00:13.636499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.636468 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:13.642309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.642283 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:13.725993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.725961 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:00:13.732104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.732035 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.755475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.754261 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-387b16-openshift-default-dockercfg-hxplc\"" Oct 30 06:00:13.760242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.760215 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:00:13.797494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797457 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.797758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.797758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797592 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.797758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797643 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.797758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797670 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.798361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.797939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.798361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.798061 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqfdg\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.798361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.798097 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.798361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.798135 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.899870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.899783 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.899870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.899846 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.899870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.899865 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.899884 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.899901 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.900296 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.900501 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.900807 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqfdg\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.900849 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.900996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.900894 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.901506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.901479 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.901774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.901741 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.901774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.901762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.901975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.901806 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.902688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.902669 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.902832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.902815 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.946139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.946112 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:13.948094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:13.948068 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqfdg\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:14.059080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.054504 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:14.324126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.324094 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:14.328867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.328773 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:00:14.664344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.663080 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:00:14.664344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.664219 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:14.664344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.664296 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:14.664344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:14.664336 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:15.323898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.323844 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerStarted","Data":"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1"} Oct 30 06:00:15.324123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.323915 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerStarted","Data":"4ef7e09f533a6ee7443edb5fafbe037bf45b31daf911ab711b336ed55e1055f0"} Oct 30 06:00:15.395719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.395664 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podStartSLOduration=2.395648048 podStartE2EDuration="2.395648048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:15.388404763 +0000 UTC m=+1673.828781939" watchObservedRunningTime="2026-10-30 06:00:15.395648048 +0000 UTC m=+1673.836025240" Oct 30 06:00:15.676140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.668399 6171 scope.go:122] "RemoveContainer" containerID="3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7" Oct 30 06:00:15.684675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.684639 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:00:15.695764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.695727 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:15.704186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.701038 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:00:15.712915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.712885 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c1d09f\"" Oct 30 06:00:15.763776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.763738 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6rbv\" (UniqueName: \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:15.763972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.763860 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:15.893068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.882738 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6rbv\" (UniqueName: \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:15.893068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.882803 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:15.893068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:15.883884 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:16.070096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.053343 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6rbv\" (UniqueName: \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\") pod \"payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:16.070096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.059328 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:16.160097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.154354 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:16.383418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.383322 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:16.395944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.395911 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerStarted","Data":"27ff6fdc80cea5932f9602e78acdc6c217e24ba45f0b0036e2d40b83a32c253f"} Oct 30 06:00:16.396473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.396343 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:16.397621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.397602 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:00:16.664839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.664761 6171 scope.go:122] "RemoveContainer" containerID="99999aea9d937639b1929f201e3cdacd2b8aa40acafb64109554ba3fff8d36f2" Oct 30 06:00:16.665549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.665523 6171 scope.go:122] "RemoveContainer" containerID="9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a" Oct 30 06:00:16.666445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.666114 6171 scope.go:122] "RemoveContainer" containerID="81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36" Oct 30 06:00:16.743114 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:16.743081 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ccb28b_0899_4d2d_8f64_a66407b67dee.slice/crio-e48bb76769069dfec26f96e751a1ea3f5ec8d5eceaf614ee36e1998f7dc206cf WatchSource:0}: Error finding container e48bb76769069dfec26f96e751a1ea3f5ec8d5eceaf614ee36e1998f7dc206cf: Status 404 returned error can't find the container with id e48bb76769069dfec26f96e751a1ea3f5ec8d5eceaf614ee36e1998f7dc206cf Oct 30 06:00:16.744438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:16.744395 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:00:17.407290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.407246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerStarted","Data":"8d9e123c254511b31b7539387b2e754af08088ca284119bfd4635523e7801ccd"} Oct 30 06:00:17.409190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.409161 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" event={"ID":"e9ccb28b-0899-4d2d-8f64-a66407b67dee","Type":"ContainerStarted","Data":"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946"} Oct 30 06:00:17.409313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.409211 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:17.409313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.409228 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" event={"ID":"e9ccb28b-0899-4d2d-8f64-a66407b67dee","Type":"ContainerStarted","Data":"e48bb76769069dfec26f96e751a1ea3f5ec8d5eceaf614ee36e1998f7dc206cf"} Oct 30 06:00:17.411339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.411317 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerStarted","Data":"af578028e754905520f4ca3de0a12137a5f11a6e08a0d7e4ef416544259d23db"} Oct 30 06:00:17.413383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.413357 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerStarted","Data":"9a31e78fc7c7163eca30951a79ae9060718f3f9d5a53e277ed555fbf160b7ad6"} Oct 30 06:00:17.415403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.415381 6171 generic.go:356] "Generic (PLEG): container finished" podID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerID="27ff6fdc80cea5932f9602e78acdc6c217e24ba45f0b0036e2d40b83a32c253f" exitCode=60 Oct 30 06:00:17.415490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.415448 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerDied","Data":"27ff6fdc80cea5932f9602e78acdc6c217e24ba45f0b0036e2d40b83a32c253f"} Oct 30 06:00:17.415550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.415498 6171 scope.go:122] "RemoveContainer" containerID="3068d89d07cdae53926087b25151d0a6ae9a08973fd4ee0a5d77e0cad654b7c7" Oct 30 06:00:17.415918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.415889 6171 scope.go:122] "RemoveContainer" containerID="27ff6fdc80cea5932f9602e78acdc6c217e24ba45f0b0036e2d40b83a32c253f" Oct 30 06:00:17.416222 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:17.416193 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276_odh-ai-gateway-infra(8814ac67-80a5-44d0-b25c-5751cc589ccf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" Oct 30 06:00:17.464170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.464143 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:17.671078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.670653 6171 scope.go:122] "RemoveContainer" containerID="87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3" Oct 30 06:00:17.678417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.674734 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" podStartSLOduration=2.674715489 podStartE2EDuration="2.674715489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:17.652770916 +0000 UTC m=+1676.093148110" watchObservedRunningTime="2026-10-30 06:00:17.674715489 +0000 UTC m=+1676.115092685" Oct 30 06:00:17.698925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.690563 6171 scope.go:122] "RemoveContainer" containerID="091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93" Oct 30 06:00:17.719104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.719082 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:00:17.740961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.740900 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:17.743354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.743330 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:00:17.760924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.760895 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-aee070\"" Oct 30 06:00:17.771454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.770363 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:17.776906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.776883 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:00:17.792583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.789739 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:17.806411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.806302 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-aee070-dockercfg-566v5\"" Oct 30 06:00:17.806411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:17.806348 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:00:18.027477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.002797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k67rq\" (UniqueName: \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.027477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.002871 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x46m5\" (UniqueName: \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.027477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.003105 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.027477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.003668 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.120357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.116911 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k67rq\" (UniqueName: \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.120357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.116963 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x46m5\" (UniqueName: \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.120357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.117001 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.120357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.117195 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.120357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.118288 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.153070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.151549 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.165393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.165364 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:00:18.165547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.165421 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:18.225834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.225805 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 06:00:18.285516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.285167 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x46m5\" (UniqueName: \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\") pod \"payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.285590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.285556 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k67rq\" (UniqueName: \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\") pod \"payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.319389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319335 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.319535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319436 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.319535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319466 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.319629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319615 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319756 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319764 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319848 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319900 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319960 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320068 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\") pod \"10899e08-3645-41ac-aafe-3f18c4684ccd\" (UID: \"10899e08-3645-41ac-aafe-3f18c4684ccd\") " Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.319953 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.320121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320090 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.320495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320204 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.320495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320242 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.320793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320770 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.320884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320799 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.320884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320813 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.320884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320829 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10899e08-3645-41ac-aafe-3f18c4684ccd-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.320884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.320842 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.326465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.326437 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "kube-api-access-72jzf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.327465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.327357 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.328796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.328758 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:18.330794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.330767 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token" pod "10899e08-3645-41ac-aafe-3f18c4684ccd" (UID: "10899e08-3645-41ac-aafe-3f18c4684ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.359457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.359436 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421427 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421491 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421531 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421607 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421683 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421799 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.422996 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.421837 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.423447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.423558 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\") pod \"63894899-4069-4607-824b-71d0ddaa304e\" (UID: \"63894899-4069-4607-824b-71d0ddaa304e\") " Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425464 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425494 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72jzf\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-kube-api-access-72jzf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425509 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425522 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425535 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10899e08-3645-41ac-aafe-3f18c4684ccd-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.425746 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.426887 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.426924 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.427191 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.428122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.427231 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.432702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.432674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:18.432702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.432682 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "kube-api-access-427hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.434362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.434332 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token" pod "63894899-4069-4607-824b-71d0ddaa304e" (UID: "63894899-4069-4607-824b-71d0ddaa304e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.435174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.435153 6171 generic.go:356] "Generic (PLEG): container finished" podID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerID="af578028e754905520f4ca3de0a12137a5f11a6e08a0d7e4ef416544259d23db" exitCode=60 Oct 30 06:00:18.435260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.435226 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerDied","Data":"af578028e754905520f4ca3de0a12137a5f11a6e08a0d7e4ef416544259d23db"} Oct 30 06:00:18.435321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.435293 6171 scope.go:122] "RemoveContainer" containerID="81090967f3298da984eed56cf787a3ca0c685566bff50551d4a2fa23da6bdf36" Oct 30 06:00:18.435776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.435755 6171 scope.go:122] "RemoveContainer" containerID="af578028e754905520f4ca3de0a12137a5f11a6e08a0d7e4ef416544259d23db" Oct 30 06:00:18.436429 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:18.436290 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k_odh-ai-gateway-infra(91c5ed71-35d2-441a-a2b3-f3332c7c0980)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" Oct 30 06:00:18.437864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.437840 6171 generic.go:356] "Generic (PLEG): container finished" podID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerID="9a31e78fc7c7163eca30951a79ae9060718f3f9d5a53e277ed555fbf160b7ad6" exitCode=60 Oct 30 06:00:18.437966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.437930 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerDied","Data":"9a31e78fc7c7163eca30951a79ae9060718f3f9d5a53e277ed555fbf160b7ad6"} Oct 30 06:00:18.445030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.445003 6171 generic.go:356] "Generic (PLEG): container finished" podID="63894899-4069-4607-824b-71d0ddaa304e" containerID="6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957" exitCode=0 Oct 30 06:00:18.445198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.445177 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" event={"ID":"63894899-4069-4607-824b-71d0ddaa304e","Type":"ContainerDied","Data":"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957"} Oct 30 06:00:18.445327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.445229 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" event={"ID":"63894899-4069-4607-824b-71d0ddaa304e","Type":"ContainerDied","Data":"8f213ccf7e11aa92548f6c6a1804d46a78ad6e06e24a1a2acf938595a7d87ce3"} Oct 30 06:00:18.445327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.445309 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z" Oct 30 06:00:18.450720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.450695 6171 generic.go:356] "Generic (PLEG): container finished" podID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerID="8d9e123c254511b31b7539387b2e754af08088ca284119bfd4635523e7801ccd" exitCode=60 Oct 30 06:00:18.450841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.450822 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerDied","Data":"8d9e123c254511b31b7539387b2e754af08088ca284119bfd4635523e7801ccd"} Oct 30 06:00:18.451306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.451287 6171 scope.go:122] "RemoveContainer" containerID="8d9e123c254511b31b7539387b2e754af08088ca284119bfd4635523e7801ccd" Oct 30 06:00:18.451704 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:18.451683 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch_odh-ai-gateway-infra(78d3f6b6-deef-4822-bac3-1adc7221f4f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" Oct 30 06:00:18.458091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.454396 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" Oct 30 06:00:18.465744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.465710 6171 generic.go:356] "Generic (PLEG): container finished" podID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerID="c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46" exitCode=0 Oct 30 06:00:18.468260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.468219 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerDied","Data":"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46"} Oct 30 06:00:18.468375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.468277 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" event={"ID":"10899e08-3645-41ac-aafe-3f18c4684ccd","Type":"ContainerDied","Data":"c33e646e5c78fa722431716e3dd7f94b1e8cbb099dff8c594808001cb9409cfc"} Oct 30 06:00:18.472424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.472399 6171 scope.go:122] "RemoveContainer" containerID="99999aea9d937639b1929f201e3cdacd2b8aa40acafb64109554ba3fff8d36f2" Oct 30 06:00:18.478783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.478761 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:18.497468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.497321 6171 scope.go:122] "RemoveContainer" containerID="6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957" Oct 30 06:00:18.516306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.516118 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:18.518399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.518379 6171 scope.go:122] "RemoveContainer" containerID="6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957" Oct 30 06:00:18.518718 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:18.518697 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957\": container with ID starting with 6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957 not found: ID does not exist" containerID="6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957" Oct 30 06:00:18.518809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.518732 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957"} err="failed to get container status \"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957\": rpc error: code = NotFound desc = could not find container \"6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957\": container with ID starting with 6390b25232801cff92c1680757fee88d1b4cd41718276bebdaeb581ef71b4957 not found: ID does not exist" Oct 30 06:00:18.518809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.518754 6171 scope.go:122] "RemoveContainer" containerID="9c23b2a2ced79d1f5dbef07e27dc14e587bb39e91862a8a60f97a1943df7ac0a" Oct 30 06:00:18.522994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.522801 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528156 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/63894899-4069-4607-824b-71d0ddaa304e-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528267 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-427hg\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-kube-api-access-427hg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528294 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/63894899-4069-4607-824b-71d0ddaa304e-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528311 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528327 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528342 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528355 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/63894899-4069-4607-824b-71d0ddaa304e-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.528746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.528368 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/63894899-4069-4607-824b-71d0ddaa304e-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.549578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.549537 6171 scope.go:122] "RemoveContainer" containerID="c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46" Oct 30 06:00:18.561444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.561426 6171 scope.go:122] "RemoveContainer" containerID="f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a" Oct 30 06:00:18.573055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.573017 6171 scope.go:122] "RemoveContainer" containerID="c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46" Oct 30 06:00:18.573388 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:18.573368 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46\": container with ID starting with c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46 not found: ID does not exist" containerID="c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46" Oct 30 06:00:18.573457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.573397 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46"} err="failed to get container status \"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46\": rpc error: code = NotFound desc = could not find container \"c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46\": container with ID starting with c71ed2c15505bd46307fb1a221440e52e84a06edc0210b07c7ff1bb8c9bc0d46 not found: ID does not exist" Oct 30 06:00:18.573457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.573417 6171 scope.go:122] "RemoveContainer" containerID="f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a" Oct 30 06:00:18.573743 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:18.573714 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a\": container with ID starting with f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a not found: ID does not exist" containerID="f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a" Oct 30 06:00:18.573798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.573742 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a"} err="failed to get container status \"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a\": rpc error: code = NotFound desc = could not find container \"f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a\": container with ID starting with f267d588ab48e20aedd89f7891c099925698c999f908a7bc9314fac3a03caf8a not found: ID does not exist" Oct 30 06:00:18.691417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.673844 6171 scope.go:122] "RemoveContainer" containerID="53b0a85d2b28b034ad349a74d0b96f4493db54028e8dd228dbed9c2e6b72e527" Oct 30 06:00:18.706157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.705516 6171 patch_prober.go:29] interesting pod/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:00:18.706157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.705574 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:00:18.841780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:18.841739 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:19.077405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.066627 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 06:00:19.078900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.078776 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-q5n2z"] Oct 30 06:00:19.096866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.096837 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 06:00:19.114127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.114100 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4fc75f-openshift-default-5867848d5c-vlwms"] Oct 30 06:00:19.123828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.123802 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.124824 6171 cpu_manager.go:414] "containerMap: removing container" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.124846 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.124979 6171 cpu_manager.go:414] "containerMap: removing container" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.124987 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.125021 6171 cpu_manager.go:414] "containerMap: removing container" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.125030 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.125241 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.125259 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" containerName="istio-proxy" Oct 30 06:00:19.125469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.125268 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="63894899-4069-4607-824b-71d0ddaa304e" containerName="istio-proxy" Oct 30 06:00:19.148093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.143313 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:00:19.148093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.143429 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.160187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.160166 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:19.161564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.161545 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-f78ca5-dockercfg-4hlbq\"" Oct 30 06:00:19.163064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.163027 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-f78ca5\"" Oct 30 06:00:19.195270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.195241 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:00:19.196774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.196751 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.196882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.196777 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.196882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.196809 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.196882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.196817 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.197152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.197132 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.197230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.197177 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:19.207262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.207232 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\" (UniqueName: \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\") pod \"8814ac67-80a5-44d0-b25c-5751cc589ccf\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " Oct 30 06:00:19.207544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.207520 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\") pod \"8814ac67-80a5-44d0-b25c-5751cc589ccf\" (UID: \"8814ac67-80a5-44d0-b25c-5751cc589ccf\") " Oct 30 06:00:19.207995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.207966 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.208228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.207989 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca" pod "8814ac67-80a5-44d0-b25c-5751cc589ccf" (UID: "8814ac67-80a5-44d0-b25c-5751cc589ccf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:19.208327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.208305 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmfv\" (UniqueName: \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.208651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.208630 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8814ac67-80a5-44d0-b25c-5751cc589ccf-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.210559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.210534 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.210649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.210610 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm" pod "8814ac67-80a5-44d0-b25c-5751cc589ccf" (UID: "8814ac67-80a5-44d0-b25c-5751cc589ccf"). InnerVolumeSpecName "kube-api-access-5q6vm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:19.221966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.221941 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:00:19.386083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.380589 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmfv\" (UniqueName: \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.386083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.380706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqcfx\" (UniqueName: \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.386083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.380745 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.386083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.380875 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.446154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.381162 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q6vm\" (UniqueName: \"kubernetes.io/projected/8814ac67-80a5-44d0-b25c-5751cc589ccf-kube-api-access-5q6vm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.535797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.396514 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.556382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.553375 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:19.597632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.562625 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqcfx\" (UniqueName: \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.597632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.562679 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.609156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.607434 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.669312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.663562 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmfv\" (UniqueName: \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\") pod \"payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.669312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.665661 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\") pod \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " Oct 30 06:00:19.669312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.665699 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\" (UniqueName: \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\") pod \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\" (UID: \"64a1b9f6-c4d4-433e-83d0-742c0a406ed4\") " Oct 30 06:00:19.683954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.670193 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca" pod "64a1b9f6-c4d4-433e-83d0-742c0a406ed4" (UID: "64a1b9f6-c4d4-433e-83d0-742c0a406ed4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:19.683954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.676967 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqcfx\" (UniqueName: \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\") pod \"payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.689405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.685228 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx" pod "64a1b9f6-c4d4-433e-83d0-742c0a406ed4" (UID: "64a1b9f6-c4d4-433e-83d0-742c0a406ed4"). InnerVolumeSpecName "kube-api-access-g5wxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:19.690247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.689721 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10899e08-3645-41ac-aafe-3f18c4684ccd" path="/var/lib/kubelet/pods/10899e08-3645-41ac-aafe-3f18c4684ccd/volumes" Oct 30 06:00:19.690616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.690510 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63894899-4069-4607-824b-71d0ddaa304e" path="/var/lib/kubelet/pods/63894899-4069-4607-824b-71d0ddaa304e/volumes" Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.690958 6171 generic.go:356] "Generic (PLEG): container finished" podID="0becf319-257f-4b48-87d6-788e15435d0e" containerID="57366671e2bce343cdb4d9ea03f3984582b17cbdcaaa4855fc0ac3c7bf40c7ab" exitCode=60 Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.691483 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerDied","Data":"57366671e2bce343cdb4d9ea03f3984582b17cbdcaaa4855fc0ac3c7bf40c7ab"} Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.691546 6171 scope.go:122] "RemoveContainer" containerID="091ab5f1243ce0d705d8237c301e0a1e80863e024bce327ac9ea6036a86acc93" Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.704585 6171 scope.go:122] "RemoveContainer" containerID="57366671e2bce343cdb4d9ea03f3984582b17cbdcaaa4855fc0ac3c7bf40c7ab" Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:19.704927 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t_odh-ai-gateway-infra(0becf319-257f-4b48-87d6-788e15435d0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" podUID="0becf319-257f-4b48-87d6-788e15435d0e" Oct 30 06:00:19.707174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.705433 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.716115 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.722485 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.722847 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" event={"ID":"64a1b9f6-c4d4-433e-83d0-742c0a406ed4","Type":"ContainerDied","Data":"1443fd454848eeab03651d114eb731b3fe053a93afc6d51c41821567f3515af1"} Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.722859 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5" Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.724930 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" event={"ID":"8814ac67-80a5-44d0-b25c-5751cc589ccf","Type":"ContainerDied","Data":"1343a432c9fae7b054568ef1e2006cd1b67db3dc469a014c39579fd61b58308e"} Oct 30 06:00:19.726288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.725297 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276" Oct 30 06:00:19.734370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.733037 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerStarted","Data":"eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584"} Oct 30 06:00:19.734370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.733559 6171 scope.go:122] "RemoveContainer" containerID="eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584" Oct 30 06:00:19.734370 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:19.733932 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm_odh-ai-gateway-infra(5f7922d7-702a-4a0a-a32a-fa8017c00540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" Oct 30 06:00:19.759009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.758883 6171 scope.go:122] "RemoveContainer" containerID="9a31e78fc7c7163eca30951a79ae9060718f3f9d5a53e277ed555fbf160b7ad6" Oct 30 06:00:19.771034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.768474 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.771034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.768499 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5wxx\" (UniqueName: \"kubernetes.io/projected/64a1b9f6-c4d4-433e-83d0-742c0a406ed4-kube-api-access-g5wxx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.783970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.783943 6171 scope.go:122] "RemoveContainer" containerID="27ff6fdc80cea5932f9602e78acdc6c217e24ba45f0b0036e2d40b83a32c253f" Oct 30 06:00:19.936770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.936738 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:19.943821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.943794 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889000-cg9b5"] Oct 30 06:00:19.946445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.946423 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:19.962813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:19.962743 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:20.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.078499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\") pod \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " Oct 30 06:00:20.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.078583 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\" (UniqueName: \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\") pod \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\" (UID: \"91c5ed71-35d2-441a-a2b3-f3332c7c0980\") " Oct 30 06:00:20.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.095025 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca" pod "91c5ed71-35d2-441a-a2b3-f3332c7c0980" (UID: "91c5ed71-35d2-441a-a2b3-f3332c7c0980"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:20.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.097736 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:20.104732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.104714 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g" pod "91c5ed71-35d2-441a-a2b3-f3332c7c0980" (UID: "91c5ed71-35d2-441a-a2b3-f3332c7c0980"). InnerVolumeSpecName "kube-api-access-fx86g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:20.138390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.133113 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:00:20.145745 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.145713 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:20.155070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.155029 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-197e9b-29889000-wf276"] Oct 30 06:00:20.181925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.181893 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\" (UniqueName: \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\") pod \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " Oct 30 06:00:20.182294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.182273 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\") pod \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\" (UID: \"78d3f6b6-deef-4822-bac3-1adc7221f4f2\") " Oct 30 06:00:20.182658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.182631 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca" pod "78d3f6b6-deef-4822-bac3-1adc7221f4f2" (UID: "78d3f6b6-deef-4822-bac3-1adc7221f4f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:20.185716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.184903 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c5ed71-35d2-441a-a2b3-f3332c7c0980-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:20.185716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.184931 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx86g\" (UniqueName: \"kubernetes.io/projected/91c5ed71-35d2-441a-a2b3-f3332c7c0980-kube-api-access-fx86g\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:20.185716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.184946 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/78d3f6b6-deef-4822-bac3-1adc7221f4f2-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:20.193160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.193132 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs" pod "78d3f6b6-deef-4822-bac3-1adc7221f4f2" (UID: "78d3f6b6-deef-4822-bac3-1adc7221f4f2"). InnerVolumeSpecName "kube-api-access-k8lgs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:20.286711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.286679 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8lgs\" (UniqueName: \"kubernetes.io/projected/78d3f6b6-deef-4822-bac3-1adc7221f4f2-kube-api-access-k8lgs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:20.339539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.339512 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:00:20.342336 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:20.342314 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38c144f1_2255_40ea_926f_01b4bd35feea.slice/crio-59a47ffa8761b8965015b15a506445f44acb0d03b499ae33e939239fff5534b4 WatchSource:0}: Error finding container 59a47ffa8761b8965015b15a506445f44acb0d03b499ae33e939239fff5534b4: Status 404 returned error can't find the container with id 59a47ffa8761b8965015b15a506445f44acb0d03b499ae33e939239fff5534b4 Oct 30 06:00:20.750081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.750056 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" Oct 30 06:00:20.750081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.750032 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch" event={"ID":"78d3f6b6-deef-4822-bac3-1adc7221f4f2","Type":"ContainerDied","Data":"fef07cbe3d3f6ef24cbbe19bb20049acd6c16d25ac892acada3a3262ab1ef32e"} Oct 30 06:00:20.750631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.750145 6171 scope.go:122] "RemoveContainer" containerID="8d9e123c254511b31b7539387b2e754af08088ca284119bfd4635523e7801ccd" Oct 30 06:00:20.752146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.752017 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" event={"ID":"38c144f1-2255-40ea-926f-01b4bd35feea","Type":"ContainerStarted","Data":"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db"} Oct 30 06:00:20.752146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.752091 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" event={"ID":"38c144f1-2255-40ea-926f-01b4bd35feea","Type":"ContainerStarted","Data":"59a47ffa8761b8965015b15a506445f44acb0d03b499ae33e939239fff5534b4"} Oct 30 06:00:20.752146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.752121 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:20.754141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.754114 6171 generic.go:356] "Generic (PLEG): container finished" podID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerID="5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449" exitCode=0 Oct 30 06:00:20.754306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.754195 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerDied","Data":"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449"} Oct 30 06:00:20.754947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.754927 6171 scope.go:122] "RemoveContainer" containerID="5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449" Oct 30 06:00:20.756954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.756926 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:20.757152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.756996 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:20.757152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.757059 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:00:20.758709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.758687 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" event={"ID":"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3","Type":"ContainerStarted","Data":"a1f2adebf289318c93646b2499edc347fb74da7e7b2fc6e41b819ce2131f4bfb"} Oct 30 06:00:20.759180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.758725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" event={"ID":"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3","Type":"ContainerStarted","Data":"c967519399c1fc4d54384d11a038623e60df20c4ddbbb0a641daccfa5606e2c0"} Oct 30 06:00:20.759180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.758796 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:20.763262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.763239 6171 generic.go:356] "Generic (PLEG): container finished" podID="9fac396a-8fbd-46d7-902b-662e1629b512" containerID="bacf1529aeb864b3b5d7f4a3c6361dd1eed622ff7595cdf0eb72401cdd334145" exitCode=60 Oct 30 06:00:20.763387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.763260 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerDied","Data":"bacf1529aeb864b3b5d7f4a3c6361dd1eed622ff7595cdf0eb72401cdd334145"} Oct 30 06:00:20.763597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.763578 6171 scope.go:122] "RemoveContainer" containerID="bacf1529aeb864b3b5d7f4a3c6361dd1eed622ff7595cdf0eb72401cdd334145" Oct 30 06:00:20.763874 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:20.763841 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz_odh-ai-gateway-infra(9fac396a-8fbd-46d7-902b-662e1629b512)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" Oct 30 06:00:20.764765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.764749 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" Oct 30 06:00:20.764876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.764765 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k" event={"ID":"91c5ed71-35d2-441a-a2b3-f3332c7c0980","Type":"ContainerDied","Data":"df79760194a4247444132333c079a68edda3697a5dd21c74652b6ca7c2a3b4fb"} Oct 30 06:00:20.765284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.765261 6171 scope.go:122] "RemoveContainer" containerID="53b0a85d2b28b034ad349a74d0b96f4493db54028e8dd228dbed9c2e6b72e527" Oct 30 06:00:20.768445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.768426 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:20.778995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.778892 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" event={"ID":"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc","Type":"ContainerStarted","Data":"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb"} Oct 30 06:00:20.778995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.778938 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" event={"ID":"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc","Type":"ContainerStarted","Data":"a35cc964e026dc83d1d69ff823f467f582057fc40cbedb04c2108b681237dd21"} Oct 30 06:00:20.778995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.778965 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:20.783373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.783348 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerID="eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584" exitCode=60 Oct 30 06:00:20.783497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.783365 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerDied","Data":"eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584"} Oct 30 06:00:20.784068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.783767 6171 scope.go:122] "RemoveContainer" containerID="eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584" Oct 30 06:00:20.784281 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:20.784239 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm_odh-ai-gateway-infra(5f7922d7-702a-4a0a-a32a-fa8017c00540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" Oct 30 06:00:20.785823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.785800 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" event={"ID":"d891f4fe-f2b4-457e-970c-049dc260f29e","Type":"ContainerStarted","Data":"52d8e5ed7a2321808e7de012dbc6dc112e6f12cb5c0d0add180d824406e8be8a"} Oct 30 06:00:20.785953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.785840 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" event={"ID":"d891f4fe-f2b4-457e-970c-049dc260f29e","Type":"ContainerStarted","Data":"e923a041e02758f1d5903f8cd872f4b4fb127ff54338ca539d3e959e1cc4252c"} Oct 30 06:00:20.785953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.785885 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:20.789082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.789060 6171 scope.go:122] "RemoveContainer" containerID="af578028e754905520f4ca3de0a12137a5f11a6e08a0d7e4ef416544259d23db" Oct 30 06:00:20.806864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.805475 6171 scope.go:122] "RemoveContainer" containerID="87dd241f4ca7232916e4747dc3a64a38639a8bfd8ac910dc2afc9ba5901e81f3" Oct 30 06:00:20.882794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.882733 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" podStartSLOduration=1.882712883 podStartE2EDuration="1.882712883s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:20.873068812 +0000 UTC m=+1679.313446004" watchObservedRunningTime="2026-10-30 06:00:20.882712883 +0000 UTC m=+1679.323090078" Oct 30 06:00:20.971191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.971139 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" podStartSLOduration=3.971120145 podStartE2EDuration="3.971120145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:20.968076041 +0000 UTC m=+1679.408453228" watchObservedRunningTime="2026-10-30 06:00:20.971120145 +0000 UTC m=+1679.411497335" Oct 30 06:00:20.981700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:20.981677 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:21.006519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.003827 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:21.078844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.078814 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:21.091958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.091927 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-de7242-29889000-b5lch"] Oct 30 06:00:21.100960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.100935 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\") pod \"0becf319-257f-4b48-87d6-788e15435d0e\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " Oct 30 06:00:21.101180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.101168 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\" (UniqueName: \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\") pod \"0becf319-257f-4b48-87d6-788e15435d0e\" (UID: \"0becf319-257f-4b48-87d6-788e15435d0e\") " Oct 30 06:00:21.101343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.101319 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca" pod "0becf319-257f-4b48-87d6-788e15435d0e" (UID: "0becf319-257f-4b48-87d6-788e15435d0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:21.101674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.101657 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0becf319-257f-4b48-87d6-788e15435d0e-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.103364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.103340 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7" pod "0becf319-257f-4b48-87d6-788e15435d0e" (UID: "0becf319-257f-4b48-87d6-788e15435d0e"). InnerVolumeSpecName "kube-api-access-57jh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:21.161068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.160650 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:21.222380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.222336 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-57jh7\" (UniqueName: \"kubernetes.io/projected/0becf319-257f-4b48-87d6-788e15435d0e-kube-api-access-57jh7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.227724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.227692 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-128436-29889000-jvx7k"] Oct 30 06:00:21.299399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.299298 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" podStartSLOduration=2.299281719 podStartE2EDuration="2.299281719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:21.296438881 +0000 UTC m=+1679.736816075" watchObservedRunningTime="2026-10-30 06:00:21.299281719 +0000 UTC m=+1679.739658917" Oct 30 06:00:21.395338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.395285 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" podStartSLOduration=4.395270898 podStartE2EDuration="4.395270898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:21.393362609 +0000 UTC m=+1679.833739803" watchObservedRunningTime="2026-10-30 06:00:21.395270898 +0000 UTC m=+1679.835648091" Oct 30 06:00:21.729275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.726572 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" path="/var/lib/kubelet/pods/64a1b9f6-c4d4-433e-83d0-742c0a406ed4/volumes" Oct 30 06:00:21.729275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.727417 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" path="/var/lib/kubelet/pods/78d3f6b6-deef-4822-bac3-1adc7221f4f2/volumes" Oct 30 06:00:21.729275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.728020 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" path="/var/lib/kubelet/pods/8814ac67-80a5-44d0-b25c-5751cc589ccf/volumes" Oct 30 06:00:21.729586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.729541 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" path="/var/lib/kubelet/pods/91c5ed71-35d2-441a-a2b3-f3332c7c0980/volumes" Oct 30 06:00:21.823807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.823757 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerStarted","Data":"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8"} Oct 30 06:00:21.825438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.825417 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" Oct 30 06:00:21.825579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.825476 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t" event={"ID":"0becf319-257f-4b48-87d6-788e15435d0e","Type":"ContainerDied","Data":"7e46815afac22ab852412b2712f87f7e3787805b58203ab8e7c819ee84b92bc5"} Oct 30 06:00:21.825579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.825526 6171 scope.go:122] "RemoveContainer" containerID="57366671e2bce343cdb4d9ea03f3984582b17cbdcaaa4855fc0ac3c7bf40c7ab" Oct 30 06:00:21.844557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:21.844526 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:22.018068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.018004 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:00:22.019351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019326 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.019529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019513 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.019636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019626 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.019783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019769 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.019878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019866 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:22.019959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.019949 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:22.020114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020102 6171 cpu_manager.go:414] "containerMap: removing container" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.020212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020201 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.020301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020291 6171 cpu_manager.go:414] "containerMap: removing container" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.020379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020369 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.020481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020470 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.020553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020544 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.020661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020652 6171 cpu_manager.go:414] "containerMap: removing container" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.020762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020749 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.020857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020847 6171 cpu_manager.go:414] "containerMap: removing container" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.020931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020923 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.020998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.020992 6171 cpu_manager.go:414] "containerMap: removing container" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.021072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021062 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.021152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021144 6171 cpu_manager.go:414] "containerMap: removing container" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.021210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021203 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.021260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021254 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.021309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021302 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.021522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021512 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.021585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021579 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.021648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021642 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.021704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021698 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.021758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021753 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8814ac67-80a5-44d0-b25c-5751cc589ccf" containerName="cleanup" Oct 30 06:00:22.021813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021806 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.021867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021861 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.021920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021914 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c5ed71-35d2-441a-a2b3-f3332c7c0980" containerName="cleanup" Oct 30 06:00:22.021973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.021966 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.022027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.022021 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.028299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.028280 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.050581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.048487 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-387b16-dockercfg-n9td8\"" Oct 30 06:00:22.050581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.048622 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-387b16\"" Oct 30 06:00:22.131072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.063550 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:22.131364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.131338 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:00:22.201139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.173545 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-a-84dc9d-29889000-5nq2t"] Oct 30 06:00:22.312138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.280118 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.312138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.280368 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tthf5\" (UniqueName: \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.335107 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336647 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336670 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336703 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336710 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336896 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0becf319-257f-4b48-87d6-788e15435d0e" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336921 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="64a1b9f6-c4d4-433e-83d0-742c0a406ed4" containerName="cleanup" Oct 30 06:00:22.353294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.336937 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78d3f6b6-deef-4822-bac3-1adc7221f4f2" containerName="cleanup" Oct 30 06:00:22.374034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.374005 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.411819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.411780 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tthf5\" (UniqueName: \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.432068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.412354 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.516854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.416729 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.516854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.473579 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:00:22.516854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.494496 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tthf5\" (UniqueName: \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\") pod \"payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.615185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.615156 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:22.616376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.616358 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:22.631596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.631563 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.631796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.631775 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtmrp\" (UniqueName: \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.631889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.631783 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:00:22.632775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.632752 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:00:22.632892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.632800 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 06:00:22.655506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.655464 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.733881 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\" (UniqueName: \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\") pod \"9fac396a-8fbd-46d7-902b-662e1629b512\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.740540 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca" pod "9fac396a-8fbd-46d7-902b-662e1629b512" (UID: "9fac396a-8fbd-46d7-902b-662e1629b512"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.745595 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz" pod "9fac396a-8fbd-46d7-902b-662e1629b512" (UID: "9fac396a-8fbd-46d7-902b-662e1629b512"). InnerVolumeSpecName "kube-api-access-t2qvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.746146 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\") pod \"9fac396a-8fbd-46d7-902b-662e1629b512\" (UID: \"9fac396a-8fbd-46d7-902b-662e1629b512\") " Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.746307 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\") pod \"5f7922d7-702a-4a0a-a32a-fa8017c00540\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.746395 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\" (UniqueName: \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\") pod \"5f7922d7-702a-4a0a-a32a-fa8017c00540\" (UID: \"5f7922d7-702a-4a0a-a32a-fa8017c00540\") " Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.746739 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca" pod "5f7922d7-702a-4a0a-a32a-fa8017c00540" (UID: "5f7922d7-702a-4a0a-a32a-fa8017c00540"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.746745 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtmrp\" (UniqueName: \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.747301 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.747486 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9fac396a-8fbd-46d7-902b-662e1629b512-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.747497 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f7922d7-702a-4a0a-a32a-fa8017c00540-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.747509 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qvz\" (UniqueName: \"kubernetes.io/projected/9fac396a-8fbd-46d7-902b-662e1629b512-kube-api-access-t2qvz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.748306 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.755159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.754725 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz" pod "5f7922d7-702a-4a0a-a32a-fa8017c00540" (UID: "5f7922d7-702a-4a0a-a32a-fa8017c00540"). InnerVolumeSpecName "kube-api-access-4dfrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:22.778576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.778540 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtmrp\" (UniqueName: \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\") pod \"payload-processing-e2e-inf-a-387b16-569d556879-gmrbp\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:22.841439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.841408 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" Oct 30 06:00:22.842477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.841444 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz" event={"ID":"9fac396a-8fbd-46d7-902b-662e1629b512","Type":"ContainerDied","Data":"627b6bb4a48eca39f80c81fde8622bb1363819e45f4edc199c08596aad975d87"} Oct 30 06:00:22.842477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.841496 6171 scope.go:122] "RemoveContainer" containerID="bacf1529aeb864b3b5d7f4a3c6361dd1eed622ff7595cdf0eb72401cdd334145" Oct 30 06:00:22.845215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.845192 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" Oct 30 06:00:22.845882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.845228 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm" event={"ID":"5f7922d7-702a-4a0a-a32a-fa8017c00540","Type":"ContainerDied","Data":"2c3989e7e43b2d2a9b24abdfa49c879fc4b69f05460fdf5baae1848fe0396260"} Oct 30 06:00:22.848806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.848766 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dfrz\" (UniqueName: \"kubernetes.io/projected/5f7922d7-702a-4a0a-a32a-fa8017c00540-kube-api-access-4dfrz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.860740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:22.860717 6171 scope.go:122] "RemoveContainer" containerID="eeb3993c294bdd95135d1ae4e06ee42bb5353e51559d45045a5455a2a9954584" Oct 30 06:00:23.067821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.067625 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:23.068149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.068129 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:23.082090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.079940 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-6b5b1e-29889000-fldbz"] Oct 30 06:00:23.123479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.122380 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:23.138473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.138433 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-cc33d3-29889000-ksmkm"] Oct 30 06:00:23.144137 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:23.143614 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24543015_3a33_40ff_a581_5bd4e46b1c5b.slice/crio-51bf5d8b9257201801036fe2bfc34a6ec73af330eb149b1ead375586eecd5e0c WatchSource:0}: Error finding container 51bf5d8b9257201801036fe2bfc34a6ec73af330eb149b1ead375586eecd5e0c: Status 404 returned error can't find the container with id 51bf5d8b9257201801036fe2bfc34a6ec73af330eb149b1ead375586eecd5e0c Oct 30 06:00:23.150133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.150095 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:00:23.511213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.511175 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:00:23.632503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.632472 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:00:23.632629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.632532 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 06:00:23.676782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.676744 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0becf319-257f-4b48-87d6-788e15435d0e" path="/var/lib/kubelet/pods/0becf319-257f-4b48-87d6-788e15435d0e/volumes" Oct 30 06:00:23.677294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.677261 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" path="/var/lib/kubelet/pods/5f7922d7-702a-4a0a-a32a-fa8017c00540/volumes" Oct 30 06:00:23.677885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.677860 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" path="/var/lib/kubelet/pods/9fac396a-8fbd-46d7-902b-662e1629b512/volumes" Oct 30 06:00:23.858681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.858597 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" event={"ID":"1908ce98-e066-4e9a-a314-119aebe9b20c","Type":"ContainerStarted","Data":"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60"} Oct 30 06:00:23.858681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.858657 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" event={"ID":"1908ce98-e066-4e9a-a314-119aebe9b20c","Type":"ContainerStarted","Data":"c7af8a8105aab7c5ba398a0b15c18bd22b52ce8dbcc416e2aaeab67bf1f048fb"} Oct 30 06:00:23.859183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.858685 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:23.862628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.862603 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" event={"ID":"24543015-3a33-40ff-a581-5bd4e46b1c5b","Type":"ContainerStarted","Data":"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370"} Oct 30 06:00:23.862763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.862651 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" event={"ID":"24543015-3a33-40ff-a581-5bd4e46b1c5b","Type":"ContainerStarted","Data":"51bf5d8b9257201801036fe2bfc34a6ec73af330eb149b1ead375586eecd5e0c"} Oct 30 06:00:23.862763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.862678 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:23.912039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.911991 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" podStartSLOduration=1.9119781169999999 podStartE2EDuration="1.911978117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:23.908508279 +0000 UTC m=+1682.348885472" watchObservedRunningTime="2026-10-30 06:00:23.911978117 +0000 UTC m=+1682.352355309" Oct 30 06:00:23.977297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:23.977248 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" podStartSLOduration=2.977235203 podStartE2EDuration="2.977235203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:23.974992253 +0000 UTC m=+1682.415369449" watchObservedRunningTime="2026-10-30 06:00:23.977235203 +0000 UTC m=+1682.417612398" Oct 30 06:00:24.632520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:24.632486 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:00:24.632735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:24.632549 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 06:00:25.632412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:25.632355 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:00:25.632822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:25.632426 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 06:00:25.948822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:25.948748 6171 scope.go:122] "RemoveContainer" containerID="8090a867c2d2e22def637580ea2d05f698d1f4b82eb3df7358f57fb4a20f3b5f" Oct 30 06:00:25.960989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:25.960962 6171 scope.go:122] "RemoveContainer" containerID="5ef2be1bbb54f6ffe5f25cf90b150550e63639a0f2e1b84e523f0c74cc15ac71" Oct 30 06:00:25.972460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:25.972440 6171 scope.go:122] "RemoveContainer" containerID="f6214f98eebf8e73b93e14fc81d5e3025c6e8260ab6f1df02a57d0c3d530e9ca" Oct 30 06:00:26.409267 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:26.409231 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:00:26.635084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:26.634250 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:00:26.635084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:26.634311 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 06:00:27.637020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:27.636993 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:00:27.637496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:27.637483 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:00:27.638213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:27.638193 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:00:28.472250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:28.472217 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:00:29.718537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.718506 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:00:29.719448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719430 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719451 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719464 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719469 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719476 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719480 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719515 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719522 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719559 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719564 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719689 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719698 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719706 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719713 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.719763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.719718 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fac396a-8fbd-46d7-902b-662e1629b512" containerName="cleanup" Oct 30 06:00:29.724088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.724069 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.743457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.743431 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-f4e35c-serving-cert\"" Oct 30 06:00:29.762481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.762451 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:00:29.892660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.892624 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkdzz\" (UniqueName: \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.892863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.892701 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.893037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.893016 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.893205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.893187 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.995459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.995417 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:29.997504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.997472 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.014293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.014257 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkdzz\" (UniqueName: \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.015085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.015038 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.015845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:29.996803 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.016511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.008294 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.029435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.029404 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.062030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.061998 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkdzz\" (UniqueName: \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\") pod \"maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.340154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.340072 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:30.651562 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:30.651532 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod091ceb72_b4b7_4d26_a987_4a12ccb6710d.slice/crio-12c3af50a7e5ba67c66a8a20eefa764e682ed61ed2adfb28d1f01a919b528851 WatchSource:0}: Error finding container 12c3af50a7e5ba67c66a8a20eefa764e682ed61ed2adfb28d1f01a919b528851: Status 404 returned error can't find the container with id 12c3af50a7e5ba67c66a8a20eefa764e682ed61ed2adfb28d1f01a919b528851 Oct 30 06:00:30.652743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:30.652717 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:00:31.064577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:31.064535 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" event={"ID":"091ceb72-b4b7-4d26-a987-4a12ccb6710d","Type":"ContainerStarted","Data":"12c3af50a7e5ba67c66a8a20eefa764e682ed61ed2adfb28d1f01a919b528851"} Oct 30 06:00:31.828944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:31.828911 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:00:31.829169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:31.829003 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:00:31.829169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:31.829106 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:00:31.829673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:31.829649 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:00:32.076597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:32.076559 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" event={"ID":"091ceb72-b4b7-4d26-a987-4a12ccb6710d","Type":"ContainerStarted","Data":"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3"} Oct 30 06:00:32.076968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:32.076621 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:32.139393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:32.139238 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" podStartSLOduration=2.686039526 podStartE2EDuration="3.139199674s" totalImagesPullingTime="453.160148ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:32.123761464 +0000 UTC m=+1690.564138657" watchObservedRunningTime="2026-10-30 06:00:32.139199674 +0000 UTC m=+1690.579576867" Oct 30 06:00:34.874802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:34.874772 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:00:34.875217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:34.874849 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:00:38.092155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:38.092122 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:00:52.438125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.438090 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:00:52.439269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.439247 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:52.439269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.439269 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:52.439538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.439522 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f7922d7-702a-4a0a-a32a-fa8017c00540" containerName="cleanup" Oct 30 06:00:52.443178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.443155 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.475787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.475757 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-f78ca5-serving-cert\"" Oct 30 06:00:52.511136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.511104 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:00:52.534234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.534185 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.534471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.534448 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.534595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.534572 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zjhw\" (UniqueName: \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.534696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.534681 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.688072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.651287 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.688609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.688546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.696136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.688809 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zjhw\" (UniqueName: \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.708066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.696308 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.708066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.697280 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.708066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.684644 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.708066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.701006 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.744717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.744652 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zjhw\" (UniqueName: \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\") pod \"maas-api-e2e-api-b-f78ca5-644d695967-6bll9\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:52.767695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:52.767641 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:53.228143 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:00:53.228111 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ca92b1_aeed_41e3_bfc3_2f63026b90c1.slice/crio-79ced74400f342668602c410b413b5eaf957c91079a9c2e33f47c3e494eddf62 WatchSource:0}: Error finding container 79ced74400f342668602c410b413b5eaf957c91079a9c2e33f47c3e494eddf62: Status 404 returned error can't find the container with id 79ced74400f342668602c410b413b5eaf957c91079a9c2e33f47c3e494eddf62 Oct 30 06:00:53.229259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:53.229227 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:00:53.611749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:53.611672 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" event={"ID":"03ca92b1-aeed-41e3-bfc3-2f63026b90c1","Type":"ContainerStarted","Data":"79ced74400f342668602c410b413b5eaf957c91079a9c2e33f47c3e494eddf62"} Oct 30 06:00:54.625542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:54.625504 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" event={"ID":"03ca92b1-aeed-41e3-bfc3-2f63026b90c1","Type":"ContainerStarted","Data":"fc386c63cbb20e4ddbf68faf0f53d3696fb712d97900a1af5b6c7d488b7db94e"} Oct 30 06:00:54.626023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:54.625573 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:00:54.678340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:00:54.678276 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" podStartSLOduration=2.2891847690000002 podStartE2EDuration="2.678257514s" totalImagesPullingTime="389.072745ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:54.674868069 +0000 UTC m=+1713.115245264" watchObservedRunningTime="2026-10-30 06:00:54.678257514 +0000 UTC m=+1713.118634708" Oct 30 06:00:56.384300 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:00:56.384247 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:01:00.645824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.645797 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:01:00.796568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.796526 6171 generic.go:356] "Generic (PLEG): container finished" podID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerID="26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c" exitCode=0 Oct 30 06:01:00.796739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.796587 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerDied","Data":"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c"} Oct 30 06:01:00.797006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.796992 6171 scope.go:122] "RemoveContainer" containerID="26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c" Oct 30 06:01:00.798993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.798961 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:00.799119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.799037 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:00.799119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:00.799094 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:01.817541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:01.817449 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerStarted","Data":"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401"} Oct 30 06:01:02.639312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:02.639282 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:01:02.639507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:02.639483 6171 patch_prober.go:29] interesting pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:01:02.639561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:02.639527 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 06:01:03.650067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:03.646424 6171 patch_prober.go:29] interesting pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:01:03.650067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:03.646517 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 06:01:04.639673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:04.639591 6171 patch_prober.go:29] interesting pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:01:04.639673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:04.639647 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 06:01:05.639941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:05.639911 6171 patch_prober.go:29] interesting pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:01:05.640462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:05.639975 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 06:01:06.639303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:06.639266 6171 patch_prober.go:29] interesting pod/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:01:06.639488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:06.639318 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.216:15021/healthz/ready\": dial tcp 10.133.0.216:15021: connect: connection refused" Oct 30 06:01:07.646090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:07.646027 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:01:07.646611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:07.646562 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:01:07.647324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:07.647301 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:01:22.422693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.422659 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:01:22.426539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.426521 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.455480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.455455 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:01:22.466246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.466221 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-3cc570\"" Oct 30 06:01:22.466503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.466486 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-3cc570-dockercfg-m9gn7\"" Oct 30 06:01:22.600087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.589408 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxbx\" (UniqueName: \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.600087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.589497 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.724192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.718378 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxbx\" (UniqueName: \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.724192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.718460 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.742115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.742084 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:22.790140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:22.788800 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxbx\" (UniqueName: \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\") pod \"payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:23.040727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:23.040647 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:23.453303 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:23.453263 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62aee73c_f5fa_4579_91ff_b1f69df57228.slice/crio-7c041cafc011b55b03ea5c86c83bba8205296d47bdff3b79e05cd36da2c83552 WatchSource:0}: Error finding container 7c041cafc011b55b03ea5c86c83bba8205296d47bdff3b79e05cd36da2c83552: Status 404 returned error can't find the container with id 7c041cafc011b55b03ea5c86c83bba8205296d47bdff3b79e05cd36da2c83552 Oct 30 06:01:23.456682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:23.456648 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:01:24.154309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:24.154271 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" event={"ID":"62aee73c-f5fa-4579-91ff-b1f69df57228","Type":"ContainerStarted","Data":"7d6d776a6a1cc63c5ed86ed935d2bdbcdc9050cb5b6d3328ae309d88a78d4e2b"} Oct 30 06:01:24.154538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:24.154335 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" event={"ID":"62aee73c-f5fa-4579-91ff-b1f69df57228","Type":"ContainerStarted","Data":"7c041cafc011b55b03ea5c86c83bba8205296d47bdff3b79e05cd36da2c83552"} Oct 30 06:01:24.155181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:24.155161 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:24.226307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:24.226259 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" podStartSLOduration=2.226245972 podStartE2EDuration="2.226245972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:24.222144774 +0000 UTC m=+1742.662521967" watchObservedRunningTime="2026-10-30 06:01:24.226245972 +0000 UTC m=+1742.666623164" Oct 30 06:01:26.382731 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:01:26.382693 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:01:29.289475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:29.289440 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:01:32.361071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:32.360126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": context deadline exceeded" Oct 30 06:01:32.663128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:32.661176 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.690071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:32.689127 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:32.704066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:32.703538 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.049094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.028158 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.049094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.035383 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.116277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.065450 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:01:35.406130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.397349 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.491087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.478464 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.659710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.659608 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.660316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.660284 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:35.660787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.660763 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:35.720238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.720196 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:35.968832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:35.968729 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.042172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.042126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.042996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.042968 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:01:36.082337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.082200 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.085409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.083802 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:01:36.094065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.087335 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.094065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.087930 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:01:36.413539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.413470 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.453858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.413846 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:01:36.453858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.446298 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.710166 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.710513 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.710805 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.711022 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.712143 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.712222 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:01:36.737191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:36.712278 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:01:37.043429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.043338 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.425298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.058113 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.435288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.093499 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.451894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.130480 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.613559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.447760 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:01:37.613998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.448210 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.614370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.461190 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.722174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.719903 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" podUID="3e622128-5588-4764-91dd-f7759834b7eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.735118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.731155 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:37.735118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.731336 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:37.775120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.773273 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.824226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.824173 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:37.871597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:37.871535 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:01:38.175919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.175811 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-q74rp" podUID="3e622128-5588-4764-91dd-f7759834b7eb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.37:8000/health\": context deadline exceeded" Oct 30 06:01:38.303550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.303515 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:01:38.426271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.426178 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 06:01:38.436573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.436534 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:38.491877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.491844 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:01:38.535170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.535113 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:38.541235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.541152 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:01:38.560833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.560800 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:01:38.618074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.614507 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:38.798662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.798598 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": context deadline exceeded" Oct 30 06:01:38.799141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.798967 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:38.846066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.845999 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:38.847151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.847126 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:01:38.886072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.873740 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:38.902578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:38.888786 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:39.090163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.087983 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:01:39.161097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.159109 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:01:39.444117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.437585 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.486071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.485799 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-62316c-openshift-default-dockercfg-t276w\"" Oct 30 06:01:39.530006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.529975 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:01:39.536199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.536163 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:39.614839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.614785 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:39.616879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.616831 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:39.633571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.633532 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.634232 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636237 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636276 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636472 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.636798 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.670065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.641117 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqzgc\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:39.810070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.805138 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:40.200154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.823967 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.212286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.212251 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.212636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.212611 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.229456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.229399 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.229841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.916243 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:01:40.236145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.235467 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqzgc\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.236145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.235540 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.236145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.235594 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.236145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.235739 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.236145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.235771 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.290462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.245453 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.290462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.839457 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.290462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.883859 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:40.290462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.088443 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": context deadline exceeded" Oct 30 06:01:40.290462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:39.916187 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:40.337132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.331802 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.341066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.337343 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.459817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.459740 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqzgc\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.464505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.462075 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.469511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.467038 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.477768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.472906 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.477768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.474527 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\") pod \"e2e-inf-b-62316c-openshift-default-96996f9c9-q792b\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.543675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.543640 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:01:40.543936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.543921 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:01:40.553073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.551278 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:40.554241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.553232 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:40.633189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.633139 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:40.667397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.665667 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:01:40.784073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.753463 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:01:40.786515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.786485 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.761028 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.819953 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.820001 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.820023 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6"] Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.820064 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1135c258-llwn6"] Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.820109 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:01:40.834995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.829183 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.844180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.842708 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:01:40.858118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.857781 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:01:40.872209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.872173 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.873575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.873544 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-c54211-openshift-default-dockercfg-t6b72\"" Oct 30 06:01:40.874021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.873552 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:01:40.878877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.878850 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879029 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879208 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qfrv\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879343 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879565 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879680 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.879917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.879878 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:40.907967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:40.880199 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.016567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.016529 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.020158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.020117 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.020598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.020575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.020788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.020773 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.020920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.020906 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qfrv\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.021088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.021072 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.021494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.021475 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.021632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.021615 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.021849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.021834 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.025248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.023463 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.025248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.024715 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.025248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.024908 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.025505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.025483 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.025587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.025557 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.031187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.029434 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.031924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.031856 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.124230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.123810 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.128123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.127929 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qfrv\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\") pod \"e2e-inf-b-c54211-openshift-default-767566b477-nvw4t\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.171652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.171313 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:41.406765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.405088 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.406765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.405189 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.406765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.405242 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.416298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.416208 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:01:41.629187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.629129 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:01:41.641943 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:41.641887 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda91bdae9_a4d3_4b3d_aa6f_2720e391dc50.slice/crio-7fda8c1ffeb98f16da5bd2637206b9dc60d6613b999382343ad6f18b498a8608 WatchSource:0}: Error finding container 7fda8c1ffeb98f16da5bd2637206b9dc60d6613b999382343ad6f18b498a8608: Status 404 returned error can't find the container with id 7fda8c1ffeb98f16da5bd2637206b9dc60d6613b999382343ad6f18b498a8608 Oct 30 06:01:41.647857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.647819 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.647982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.647909 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.647982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.647954 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:41.711069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:41.706008 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a1ae32d-416f-43fe-b716-f6508e2ac261" path="/var/lib/kubelet/pods/4a1ae32d-416f-43fe-b716-f6508e2ac261/volumes" Oct 30 06:01:42.274159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.273167 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerStarted","Data":"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477"} Oct 30 06:01:42.274159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.273235 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerStarted","Data":"6d51791e6431efed2bf3e70c1d7728d87f11c404330ebddf8679c8cb33976f36"} Oct 30 06:01:42.296629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.285697 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerStarted","Data":"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5"} Oct 30 06:01:42.296629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.285759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerStarted","Data":"7fda8c1ffeb98f16da5bd2637206b9dc60d6613b999382343ad6f18b498a8608"} Oct 30 06:01:42.386346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.386283 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podStartSLOduration=6.386262403 podStartE2EDuration="6.386262403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:42.363729394 +0000 UTC m=+1760.804106588" watchObservedRunningTime="2026-10-30 06:01:42.386262403 +0000 UTC m=+1760.826639601" Oct 30 06:01:42.421694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.421630 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podStartSLOduration=5.421608814 podStartE2EDuration="5.421608814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:42.42074279 +0000 UTC m=+1760.861119985" watchObservedRunningTime="2026-10-30 06:01:42.421608814 +0000 UTC m=+1760.861986009" Oct 30 06:01:42.848172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.848126 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:42.856918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:42.856890 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:43.172103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.172000 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:43.178867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.178839 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:43.241228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.241186 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:43.241421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.241251 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:43.297453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.297256 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:43.297453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.297318 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:43.298685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.298422 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:01:43.298685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:43.298473 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:01:44.143269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.143223 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:01:44.156127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.155103 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.190484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.190422 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:01:44.322439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322353 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322404 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322531 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322721 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgn4n\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322811 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322850 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.322970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322957 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.323028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.322984 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.323028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.323025 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.425957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.425850 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgn4n\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.425957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.425918 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.425960 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426116 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426152 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426235 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426495 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426543 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.426649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.426581 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.428061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.427711 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.428061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.428031 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.428410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.428386 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.428663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.428635 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.428926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.428899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.432148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.431596 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.447095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.435319 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.459321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.459252 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.461903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.461842 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgn4n\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.506037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.498875 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:44.965465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.965432 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:01:44.968506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.968124 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:44.968506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.968210 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:44.968506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:44.968260 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:45.406506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.406206 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:45.406506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.406293 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:45.504951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.496996 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" event={"ID":"4571e3a6-c10d-4b7c-bdb3-2be7276a951f","Type":"ContainerStarted","Data":"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede"} Oct 30 06:01:45.504951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.497164 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" event={"ID":"4571e3a6-c10d-4b7c-bdb3-2be7276a951f","Type":"ContainerStarted","Data":"10ea511c6da885dbc941f26903db95e8c16f0abff66dc94698e58b5e053c596a"} Oct 30 06:01:45.512173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.510169 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:45.674169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:45.673265 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podStartSLOduration=1.673243882 podStartE2EDuration="1.673243882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:45.578120451 +0000 UTC m=+1764.018497670" watchObservedRunningTime="2026-10-30 06:01:45.673243882 +0000 UTC m=+1764.113621079" Oct 30 06:01:46.717121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:46.715624 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:46.717121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:46.715680 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:47.518164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:47.517849 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:47.518164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:47.517923 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:47.596636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:47.589201 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:01:47.596636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:47.589264 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:47.725009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:47.723443 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:01:48.187111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.178533 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:01:48.187425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.187399 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:01:48.246250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.246216 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.281130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.278896 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:01:48.281130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.279035 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.289074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.289009 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-c54211\"" Oct 30 06:01:48.300952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.300923 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-c54211-dockercfg-74gcx\"" Oct 30 06:01:48.355209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.355176 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpxqz\" (UniqueName: \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.355771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.355739 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.458824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.458720 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.459175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.459141 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpxqz\" (UniqueName: \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.459634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.459605 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.459807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.459781 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn8t6\" (UniqueName: \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.460381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.460359 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.513711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.513673 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:48.513881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.513739 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:48.515338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.515314 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpxqz\" (UniqueName: \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\") pod \"payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.581924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.576569 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn8t6\" (UniqueName: \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.589533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.582395 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.619641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.617850 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.619641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.619419 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:48.749576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.749540 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn8t6\" (UniqueName: \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\") pod \"payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:48.977148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:48.976311 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:49.135905 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:49.135462 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e168cfd_a88d_4e77_9ee5_171c88fd1af8.slice/crio-efe923c1b7baf0e78c58c14318a4fac389dc0560df2803312164d0bc534d80be WatchSource:0}: Error finding container efe923c1b7baf0e78c58c14318a4fac389dc0560df2803312164d0bc534d80be: Status 404 returned error can't find the container with id efe923c1b7baf0e78c58c14318a4fac389dc0560df2803312164d0bc534d80be Oct 30 06:01:49.136162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:49.135988 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:01:49.337884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:49.337840 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" event={"ID":"4e168cfd-a88d-4e77-9ee5-171c88fd1af8","Type":"ContainerStarted","Data":"efe923c1b7baf0e78c58c14318a4fac389dc0560df2803312164d0bc534d80be"} Oct 30 06:01:49.372530 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:49.372501 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756413e_7f5a_4f30_b9ad_2c1903c4a3da.slice/crio-4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257 WatchSource:0}: Error finding container 4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257: Status 404 returned error can't find the container with id 4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257 Oct 30 06:01:49.375709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:49.375601 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:01:49.508357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:49.508260 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:49.508357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:49.508328 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:50.532070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.516462 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:50.532070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.516517 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.532266 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569625 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569672 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569717 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" event={"ID":"8756413e-7f5a-4f30-b9ad-2c1903c4a3da","Type":"ContainerStarted","Data":"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae"} Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569742 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" event={"ID":"8756413e-7f5a-4f30-b9ad-2c1903c4a3da","Type":"ContainerStarted","Data":"4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257"} Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569790 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569799 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" event={"ID":"4e168cfd-a88d-4e77-9ee5-171c88fd1af8","Type":"ContainerStarted","Data":"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d"} Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.569844 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.617415 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" podStartSLOduration=3.617396 podStartE2EDuration="3.617396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:50.61448238 +0000 UTC m=+1769.054859573" watchObservedRunningTime="2026-10-30 06:01:50.617396 +0000 UTC m=+1769.057773195" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.667451 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" podStartSLOduration=3.667437341 podStartE2EDuration="3.667437341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:50.663386752 +0000 UTC m=+1769.103763946" watchObservedRunningTime="2026-10-30 06:01:50.667437341 +0000 UTC m=+1769.107814533" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.722819 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723025 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723307 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723553 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723608 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723680 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723735 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgl86\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.723836 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831502 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831716 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831894 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831927 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.831966 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgl86\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.832127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.832203 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.887101 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.887478 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.891697 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.893182 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.896939 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.897395 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.908145 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.930404 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.930626 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.226460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:50.942562 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgl86\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\") pod \"e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.240722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.239469 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:51.353752 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:01:51.353708 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5963eb_38cb_45a2_9ae8_7e8a60d398da.slice/crio-conmon-07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:01:51.353918 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:01:51.353850 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5963eb_38cb_45a2_9ae8_7e8a60d398da.slice/crio-07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5963eb_38cb_45a2_9ae8_7e8a60d398da.slice/crio-conmon-07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:01:51.506145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.505580 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:51.506145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.505638 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:51.579419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.579390 6171 generic.go:356] "Generic (PLEG): container finished" podID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerID="4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1" exitCode=0 Oct 30 06:01:51.579568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.579460 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerDied","Data":"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1"} Oct 30 06:01:51.580025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.580002 6171 scope.go:122] "RemoveContainer" containerID="4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1" Oct 30 06:01:51.607917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.607879 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.608088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.607966 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.608088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.608000 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.609445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.609419 6171 generic.go:356] "Generic (PLEG): container finished" podID="ad831650-4a65-447c-9145-035bcf85c782" containerID="50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394" exitCode=0 Oct 30 06:01:51.609556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.609497 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerDied","Data":"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394"} Oct 30 06:01:51.610002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.609984 6171 scope.go:122] "RemoveContainer" containerID="50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394" Oct 30 06:01:51.612682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.612654 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.612682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.612738 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.612955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.612791 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.613456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.613435 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerID="07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9" exitCode=0 Oct 30 06:01:51.613558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.613458 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerDied","Data":"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9"} Oct 30 06:01:51.613993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.613970 6171 scope.go:122] "RemoveContainer" containerID="07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9" Oct 30 06:01:51.629645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.624326 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.629645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.624404 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.629645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.624452 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.655894 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:51.655858 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80946ee4_ec79_48b2_9ae5_cb78f1c4c26a.slice/crio-359adeeca51674ebaf957529dd31871aef6afa99413efc71dd7985203baeb729 WatchSource:0}: Error finding container 359adeeca51674ebaf957529dd31871aef6afa99413efc71dd7985203baeb729: Status 404 returned error can't find the container with id 359adeeca51674ebaf957529dd31871aef6afa99413efc71dd7985203baeb729 Oct 30 06:01:51.658568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.658547 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:01:51.660360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.660314 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.660491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.660406 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:51.660491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:51.660452 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:52.520256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.520215 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" start-of-body= Oct 30 06:01:52.520657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.520621 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": dial tcp 10.133.0.243:15021: connect: connection refused" Oct 30 06:01:52.831018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.830917 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerStarted","Data":"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d"} Oct 30 06:01:52.835656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.835626 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerStarted","Data":"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e"} Oct 30 06:01:52.840094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.840066 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" event={"ID":"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a","Type":"ContainerStarted","Data":"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32"} Oct 30 06:01:52.840279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.840258 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" event={"ID":"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a","Type":"ContainerStarted","Data":"359adeeca51674ebaf957529dd31871aef6afa99413efc71dd7985203baeb729"} Oct 30 06:01:52.843198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.843173 6171 generic.go:356] "Generic (PLEG): container finished" podID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerID="4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657" exitCode=0 Oct 30 06:01:52.843390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.843370 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerDied","Data":"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657"} Oct 30 06:01:52.844025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.844007 6171 scope.go:122] "RemoveContainer" containerID="4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657" Oct 30 06:01:52.858771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.858729 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:52.858903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.858880 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:52.858993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.858938 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:01:52.882597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:52.882563 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerStarted","Data":"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772"} Oct 30 06:01:53.065410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.064365 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:01:53.065410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.064547 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:53.065410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.064589 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:53.174187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.171749 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podStartSLOduration=3.171727314 podStartE2EDuration="3.171727314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:53.165258702 +0000 UTC m=+1771.605635895" watchObservedRunningTime="2026-10-30 06:01:53.171727314 +0000 UTC m=+1771.612104508" Oct 30 06:01:53.267072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.264623 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:01:53.271066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.270150 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:53.271066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.270203 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:53.507073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.495532 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:01:53.507073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.495662 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:53.507073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.495707 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:53.637244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.637204 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:01:53.669075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.667320 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:53.669075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:53.667383 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:54.097213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.097172 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.149485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.149444 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:54.207580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.207531 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.252131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.252099 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.252753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.252723 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:54.252944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.252649 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerStarted","Data":"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664"} Oct 30 06:01:54.255120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.255091 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:54.262089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.262067 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:54.263581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.263557 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:01:54.263781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.263449 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.309188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.309145 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.508657 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.508726 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.519779 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.520171 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.520486 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.522207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.520522 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:54.571070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.562215 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.571070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.563953 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:01:54.647822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.641933 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:54.647822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.641994 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:54.778701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.777187 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.825103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.822128 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:54.990262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:54.990210 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.123871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.123783 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.135412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.135374 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:55.221110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.221064 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.221866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.221828 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:01:55.296997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.285746 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.296997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.285824 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:55.336590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.336547 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:01:55.337237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.337210 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:01:55.472462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.472412 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/health\": context deadline exceeded" Oct 30 06:01:55.486166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.481676 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.486166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.481731 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:55.553456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.553421 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.559352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.559292 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.698082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.584198 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:01:55.701076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.584258 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:01:55.738163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.736434 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:55.738163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.736509 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:55.738990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.738965 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:01:55.777947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.777901 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.782343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.782293 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.783181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.559144 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:55.837093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.835771 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.887203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.887157 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:55.903074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:55.888557 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/health\": context deadline exceeded" Oct 30 06:01:56.024149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.014230 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:56.146102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.132177 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/health\": context deadline exceeded" Oct 30 06:01:56.175104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.163324 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:56.175104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.163384 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:56.221598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.221554 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:56.249890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.249856 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:56.250159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.250130 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:56.364257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.344275 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:56.460183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.460147 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:56.460451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.460416 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:56.519067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.518255 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:56.519067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.518330 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:56.520720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.520696 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:01:56.521326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.521309 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:01:56.552171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.552143 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:01:56.627005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.626923 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:01:57.125345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.718353 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:57.297065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.718594 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 06:01:57.301984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.785589 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:57.332411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.792006 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:57.503455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.792315 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:57.503806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.888503 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:57.504035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:56.912319 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:01:57.504196 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:01:56.967378 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:01:57.504314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.031153 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:57.504497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.336709 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": context deadline exceeded" Oct 30 06:01:57.579348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.415030 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.579704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.579674 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:57.579843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.563073 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:01:57.580501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.580476 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:01:57.580752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.580735 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:01:57.580977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.580932 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:01:57.581220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.581195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:01:57.581847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.503212 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:57.582017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.507345 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.582401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.582168 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:57.582559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.570350 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:01:57.683498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.659621 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.768285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.768176 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:57.768591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.696227 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.806879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.795330 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:57.937129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.578953 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:57.937129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:57.936900 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:58.305344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.305301 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.55:50052\" within 1s: context deadline exceeded" Oct 30 06:01:58.307562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.306321 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.327668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.324401 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:58.327668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.324456 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:58.327668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.326262 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.362468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.362408 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.368490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.368458 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:58.368738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.368712 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:58.398089 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:01:58.398060 6171 logging.go:55] [core] [Channel #6303 SubChannel #6304]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.55:50052", ServerName: "10.133.0.55:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: write tcp 10.133.0.2:43934->10.133.0.55:50052: use of closed network connection Oct 30 06:01:58.465886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.449294 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:58.465886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.449351 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:58.603136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.602999 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.623673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.603628 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.625277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.623305 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded" Oct 30 06:01:58.772133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.735978 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:58.772133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.736062 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:58.776065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.774156 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.776065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.774885 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:58.776065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.775561 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": context deadline exceeded" Oct 30 06:01:58.780069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.776393 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": context deadline exceeded" Oct 30 06:01:58.853069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.839161 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:58.853069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:58.839226 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:59.139155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.089845 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.139155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.089904 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:01:59.139155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.129553 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.139155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.129616 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:59.287587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.287553 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.287892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.287854 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:01:59.312680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.312166 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.318201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.318168 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:01:59.329297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.329252 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.426163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.414112 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:01:59.445247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.445214 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.445544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.445509 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:01:59.472095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.468216 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.549072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.524115 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.618199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.551810 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 06:01:59.618668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.618185 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:01:59.652100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.638974 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.652100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.639818 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:01:59.652100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.642607 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:01:59.652100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.642657 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:01:59.682124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.681155 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.787467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.787423 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.788028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.787995 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:01:59.789294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.789267 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.789726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.789441 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:59.898513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:01:59.888074 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 06:02:00.057071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.056277 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:02:00.057071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.056348 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:02:00.115549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.115508 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.253419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.253375 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:02:00.253610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.253442 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:02:00.326272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.325187 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.349392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.329636 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.419071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.416282 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.456200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.452418 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:02:00.456200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.452478 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:02:00.527098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.520652 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:02:00.527098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.520708 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:02:00.627675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.625133 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.652362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.652289 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.654820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.654781 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.690403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.657027 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:02:00.690403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.689482 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:02:00.690403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.687156 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.801075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.788130 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.801075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.789323 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:00.801075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.789622 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": context deadline exceeded" Oct 30 06:02:00.801075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:00.797589 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.058691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.058652 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:02:01.058966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.058936 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:02:01.278120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.242395 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:02:01.278467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.278424 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:02:01.372132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.366266 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.416728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.377742 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.416728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.398179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:02:01.417185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.417136 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:02:01.462239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.462196 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:02:01.462497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.462469 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:02:01.518074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.509587 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:02:01.518074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.509653 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:02:01.659840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.659743 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:02:01.660177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.660153 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:02:01.660462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.660065 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.712091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.711448 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.712091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.711554 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:02:01.811073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.807038 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.893236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.893184 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.904206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.904158 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:01.949341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:01.949210 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:02.044078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.038522 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": context deadline exceeded" Oct 30 06:02:02.074088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.055862 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:02:02.074088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.070590 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:02:02.134276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.134238 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:02:02.404072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.369357 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:02:02.404072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.369422 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:02:02.404072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.379986 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:02.404072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.384215 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:02:02.419429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.419387 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:02.476267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.459990 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:02:02.476267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.460155 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:02:02.547069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.530015 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:02:02.547069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.530096 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:02:02.644072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.643492 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:02:02.644072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.643554 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:02:02.678127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:02.677655 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.023100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.022763 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.062261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.043215 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:02:03.062261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.043691 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.062261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.061924 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.067860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.065838 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.067860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.066224 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:02:03.067860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.066266 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:02:03.115266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.115231 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:02:03.150011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.149140 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.266869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.266829 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 06:02:03.267286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.267254 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 06:02:03.410373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.410253 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.413421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.410938 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.450141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.450089 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:03.510353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.510298 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" start-of-body= Oct 30 06:02:03.510678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.510632 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.227:15021/healthz/ready\": dial tcp 10.133.0.227:15021: connect: connection refused" Oct 30 06:02:03.511627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.511589 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:02:03.511832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.511812 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:02:03.656076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.646020 6171 patch_prober.go:29] interesting pod/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" start-of-body= Oct 30 06:02:03.656403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.656348 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.229:15021/healthz/ready\": dial tcp 10.133.0.229:15021: connect: connection refused" Oct 30 06:02:03.681222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:03.681126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.040070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.031212 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.055112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.044100 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.055112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.047945 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:02:04.066911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.066317 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.072137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.069065 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" probeResult="failure" output="Get \"https://10.133.0.209:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.097469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.097436 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" start-of-body= Oct 30 06:02:04.144073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.139579 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.230:15021/healthz/ready\": dial tcp 10.133.0.230:15021: connect: connection refused" Oct 30 06:02:04.242135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.242089 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.382069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.380588 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:02:04.580075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.579942 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:02:04.718597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.718515 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:02:04.767513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.747957 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.767513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.763017 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:02:04.767513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.758282 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" podUID="d0573b9c-6a49-48fc-9f23-8e11e909d758" containerName="payload-processing" containerID="cri-o://136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de" gracePeriod=25 Oct 30 06:02:04.767513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.766597 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:02:04.769566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.769535 6171 patch_prober.go:29] interesting pod/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" start-of-body= Oct 30 06:02:04.769785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.769583 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.228:15021/healthz/ready\": dial tcp 10.133.0.228:15021: connect: connection refused" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.771271 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.772282 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.772473 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.772648 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.777259 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:02:04.780004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.777870 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:02:04.810336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.799763 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:02:04.810336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.802728 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:02:04.832291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.829029 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:02:04.878655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.874079 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:02:04.878655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.874139 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:02:04.878655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.874179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:02:04.878655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.876669 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:02:04.878655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.878160 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:02:04.908623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.908518 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:02:04.929467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:04.926652 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" podUID="d144f7a0-4695-49cf-bce2-711fc2f550ea" containerName="payload-pre-processing" containerID="cri-o://8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8" gracePeriod=25 Oct 30 06:02:05.064536 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.063421 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:02:05.067113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.066206 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:02:05.246925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.246896 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 06:02:05.398066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.398018 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\" (UniqueName: \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\") pod \"d0573b9c-6a49-48fc-9f23-8e11e909d758\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " Oct 30 06:02:05.398461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.398329 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\") pod \"d0573b9c-6a49-48fc-9f23-8e11e909d758\" (UID: \"d0573b9c-6a49-48fc-9f23-8e11e909d758\") " Oct 30 06:02:05.410468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.400426 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume" pod "d0573b9c-6a49-48fc-9f23-8e11e909d758" (UID: "d0573b9c-6a49-48fc-9f23-8e11e909d758"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:05.427434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.427386 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698" pod "d0573b9c-6a49-48fc-9f23-8e11e909d758" (UID: "d0573b9c-6a49-48fc-9f23-8e11e909d758"). InnerVolumeSpecName "kube-api-access-bs698". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:05.449837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.449697 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 06:02:05.507079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.502341 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\") pod \"d144f7a0-4695-49cf-bce2-711fc2f550ea\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " Oct 30 06:02:05.507079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.502444 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\" (UniqueName: \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\") pod \"d144f7a0-4695-49cf-bce2-711fc2f550ea\" (UID: \"d144f7a0-4695-49cf-bce2-711fc2f550ea\") " Oct 30 06:02:05.507079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.503427 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs698\" (UniqueName: \"kubernetes.io/projected/d0573b9c-6a49-48fc-9f23-8e11e909d758-kube-api-access-bs698\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:05.507079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.503453 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d0573b9c-6a49-48fc-9f23-8e11e909d758-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:05.507079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.505450 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume" pod "d144f7a0-4695-49cf-bce2-711fc2f550ea" (UID: "d144f7a0-4695-49cf-bce2-711fc2f550ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:05.510173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.510146 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l" pod "d144f7a0-4695-49cf-bce2-711fc2f550ea" (UID: "d144f7a0-4695-49cf-bce2-711fc2f550ea"). InnerVolumeSpecName "kube-api-access-4h48l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:05.516830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.516803 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:02:05.518546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.518519 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:02:05.605173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.605110 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d144f7a0-4695-49cf-bce2-711fc2f550ea-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:05.605173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.605140 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4h48l\" (UniqueName: \"kubernetes.io/projected/d144f7a0-4695-49cf-bce2-711fc2f550ea-kube-api-access-4h48l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:05.799717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.799686 6171 generic.go:356] "Generic (PLEG): container finished" podID="d144f7a0-4695-49cf-bce2-711fc2f550ea" containerID="8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8" exitCode=0 Oct 30 06:02:05.799992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.799746 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" Oct 30 06:02:05.800126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.799768 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" event={"ID":"d144f7a0-4695-49cf-bce2-711fc2f550ea","Type":"ContainerDied","Data":"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8"} Oct 30 06:02:05.800193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.800142 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj" event={"ID":"d144f7a0-4695-49cf-bce2-711fc2f550ea","Type":"ContainerDied","Data":"753b6f39d2b853470f82a3e1c56b0a9180316be6174e6e58024cf7df2ebf7f10"} Oct 30 06:02:05.800193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.800168 6171 scope.go:122] "RemoveContainer" containerID="8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8" Oct 30 06:02:05.802447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.802424 6171 generic.go:356] "Generic (PLEG): container finished" podID="d0573b9c-6a49-48fc-9f23-8e11e909d758" containerID="136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de" exitCode=0 Oct 30 06:02:05.802562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.802484 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" Oct 30 06:02:05.802686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.802663 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" event={"ID":"d0573b9c-6a49-48fc-9f23-8e11e909d758","Type":"ContainerDied","Data":"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de"} Oct 30 06:02:05.802762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.802734 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8" event={"ID":"d0573b9c-6a49-48fc-9f23-8e11e909d758","Type":"ContainerDied","Data":"e5b67ee8c0b3c1b7df21691258fe7e232c72315931e8c8b7013677626633d19e"} Oct 30 06:02:05.835890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.835866 6171 scope.go:122] "RemoveContainer" containerID="8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8" Oct 30 06:02:05.836431 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:02:05.836401 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8\": container with ID starting with 8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8 not found: ID does not exist" containerID="8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8" Oct 30 06:02:05.836644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.836617 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8"} err="failed to get container status \"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8\": rpc error: code = NotFound desc = could not find container \"8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8\": container with ID starting with 8bc28850e6494317a7b673aa868916a141fd143d9c4ce9cc7bf57e1bb32f26b8 not found: ID does not exist" Oct 30 06:02:05.836798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.836783 6171 scope.go:122] "RemoveContainer" containerID="136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de" Oct 30 06:02:05.856227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.856199 6171 scope.go:122] "RemoveContainer" containerID="136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de" Oct 30 06:02:05.856811 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:02:05.856781 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de\": container with ID starting with 136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de not found: ID does not exist" containerID="136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de" Oct 30 06:02:05.856967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:05.856823 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de"} err="failed to get container status \"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de\": rpc error: code = NotFound desc = could not find container \"136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de\": container with ID starting with 136aad7eb6cce1e12f1e21ba412b522ccd4e7a73fb843b84ddc9dcbb1b21b2de not found: ID does not exist" Oct 30 06:02:06.044711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:06.044674 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 06:02:06.053031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:06.053003 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-84dc9d-d6945f4c5-njwwj"] Oct 30 06:02:06.060922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:06.060897 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 06:02:06.068566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:06.068537 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-84dc9d-5657dc7c79-29jx8"] Oct 30 06:02:07.677393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.677360 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0573b9c-6a49-48fc-9f23-8e11e909d758" path="/var/lib/kubelet/pods/d0573b9c-6a49-48fc-9f23-8e11e909d758/volumes" Oct 30 06:02:07.678017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.677997 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d144f7a0-4695-49cf-bce2-711fc2f550ea" path="/var/lib/kubelet/pods/d144f7a0-4695-49cf-bce2-711fc2f550ea/volumes" Oct 30 06:02:07.848256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.848209 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2"] Oct 30 06:02:07.849275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849247 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d144f7a0-4695-49cf-bce2-711fc2f550ea" containerName="payload-pre-processing" Oct 30 06:02:07.849275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849270 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d144f7a0-4695-49cf-bce2-711fc2f550ea" containerName="payload-pre-processing" Oct 30 06:02:07.849474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849293 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d0573b9c-6a49-48fc-9f23-8e11e909d758" containerName="payload-processing" Oct 30 06:02:07.849474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849298 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0573b9c-6a49-48fc-9f23-8e11e909d758" containerName="payload-processing" Oct 30 06:02:07.849550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849499 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d144f7a0-4695-49cf-bce2-711fc2f550ea" containerName="payload-pre-processing" Oct 30 06:02:07.849550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.849511 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0573b9c-6a49-48fc-9f23-8e11e909d758" containerName="payload-processing" Oct 30 06:02:07.853140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.853118 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:07.873229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.873203 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:02:07.874849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.874820 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2"] Oct 30 06:02:07.880765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.880746 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:02:07.937070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.936979 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:07.937331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.937312 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:07.937441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:07.937422 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dljl\" (UniqueName: \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.039658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.039621 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.039837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.039713 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dljl\" (UniqueName: \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.039837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.039751 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.040526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.040503 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.042334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.042305 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.065600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.065571 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dljl\" (UniqueName: \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\") pod \"maas-api-revoke-sub-13b4d176-846c2\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.172211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.172178 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:08.491883 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:02:08.491842 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0546563a_a40f_4e32_b234_2a43c2e51b66.slice/crio-8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221 WatchSource:0}: Error finding container 8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221: Status 404 returned error can't find the container with id 8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221 Oct 30 06:02:08.493807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.493778 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2"] Oct 30 06:02:08.838621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:08.838586 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" event={"ID":"0546563a-a40f-4e32-b234-2a43c2e51b66","Type":"ContainerStarted","Data":"8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221"} Oct 30 06:02:09.849866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:09.849823 6171 generic.go:356] "Generic (PLEG): container finished" podID="0546563a-a40f-4e32-b234-2a43c2e51b66" containerID="9a74ddafa10cf2abafaba8c8533718184d5bf4253f3bf8d3466d6b90a15d4864" exitCode=0 Oct 30 06:02:09.850267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:09.849907 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" event={"ID":"0546563a-a40f-4e32-b234-2a43c2e51b66","Type":"ContainerDied","Data":"9a74ddafa10cf2abafaba8c8533718184d5bf4253f3bf8d3466d6b90a15d4864"} Oct 30 06:02:11.086539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.086513 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:11.233799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.233769 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\") pod \"0546563a-a40f-4e32-b234-2a43c2e51b66\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " Oct 30 06:02:11.234015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.234002 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\") pod \"0546563a-a40f-4e32-b234-2a43c2e51b66\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " Oct 30 06:02:11.234201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.234186 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\" (UniqueName: \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\") pod \"0546563a-a40f-4e32-b234-2a43c2e51b66\" (UID: \"0546563a-a40f-4e32-b234-2a43c2e51b66\") " Oct 30 06:02:11.234373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.234350 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca" pod "0546563a-a40f-4e32-b234-2a43c2e51b66" (UID: "0546563a-a40f-4e32-b234-2a43c2e51b66"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:11.234778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.234760 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:11.236024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.236001 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca" pod "0546563a-a40f-4e32-b234-2a43c2e51b66" (UID: "0546563a-a40f-4e32-b234-2a43c2e51b66"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:11.236281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.236262 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl" pod "0546563a-a40f-4e32-b234-2a43c2e51b66" (UID: "0546563a-a40f-4e32-b234-2a43c2e51b66"). InnerVolumeSpecName "kube-api-access-4dljl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:11.336322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.336294 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dljl\" (UniqueName: \"kubernetes.io/projected/0546563a-a40f-4e32-b234-2a43c2e51b66-kube-api-access-4dljl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:11.336322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.336319 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0546563a-a40f-4e32-b234-2a43c2e51b66-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:11.871771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.871740 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" Oct 30 06:02:11.871771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.871757 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2" event={"ID":"0546563a-a40f-4e32-b234-2a43c2e51b66","Type":"ContainerDied","Data":"8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221"} Oct 30 06:02:11.872011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:11.871798 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e32afd911682eed48c829d8f56752c0fe3ebad3013ae819eb8726b0972ce221" Oct 30 06:02:19.007070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:19.006696 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:02:21.227012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.226971 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 06:02:21.901527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.901489 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv"] Oct 30 06:02:21.902465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.902448 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0546563a-a40f-4e32-b234-2a43c2e51b66" containerName="revoke-keys" Oct 30 06:02:21.902535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.902467 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0546563a-a40f-4e32-b234-2a43c2e51b66" containerName="revoke-keys" Oct 30 06:02:21.902626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.902617 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0546563a-a40f-4e32-b234-2a43c2e51b66" containerName="revoke-keys" Oct 30 06:02:21.906849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.906831 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:21.923646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.923613 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:02:21.927488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.927460 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv"] Oct 30 06:02:21.933117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:21.933094 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:02:22.002913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.002878 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjnk2\" (UniqueName: \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.003169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.003079 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.003272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.003258 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.104561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.104524 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.104744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.104615 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjnk2\" (UniqueName: \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.104837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.104747 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.105372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.105352 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.107069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.107034 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.130334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.130310 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjnk2\" (UniqueName: \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\") pod \"maas-api-revoke-sub-edde6690-qj8wv\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.226295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.226264 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:22.542903 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:02:22.542872 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod601ada4f_cb28_4958_8ddb_4e8974621dd0.slice/crio-aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639 WatchSource:0}: Error finding container aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639: Status 404 returned error can't find the container with id aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639 Oct 30 06:02:22.544696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:22.544668 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv"] Oct 30 06:02:23.114316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:23.114231 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" event={"ID":"601ada4f-cb28-4958-8ddb-4e8974621dd0","Type":"ContainerStarted","Data":"5d825ffd07018c3cec84b9b88ec06854a8ba5ea2c94f7abf52747d71471cbb25"} Oct 30 06:02:23.114316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:23.114284 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" event={"ID":"601ada4f-cb28-4958-8ddb-4e8974621dd0","Type":"ContainerStarted","Data":"aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639"} Oct 30 06:02:23.159639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:23.159583 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" podStartSLOduration=2.159569864 podStartE2EDuration="2.159569864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:23.158472184 +0000 UTC m=+1801.598849379" watchObservedRunningTime="2026-10-30 06:02:23.159569864 +0000 UTC m=+1801.599947057" Oct 30 06:02:24.125818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:24.125778 6171 generic.go:356] "Generic (PLEG): container finished" podID="601ada4f-cb28-4958-8ddb-4e8974621dd0" containerID="5d825ffd07018c3cec84b9b88ec06854a8ba5ea2c94f7abf52747d71471cbb25" exitCode=0 Oct 30 06:02:24.126270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:24.125864 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" event={"ID":"601ada4f-cb28-4958-8ddb-4e8974621dd0","Type":"ContainerDied","Data":"5d825ffd07018c3cec84b9b88ec06854a8ba5ea2c94f7abf52747d71471cbb25"} Oct 30 06:02:25.329527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.329496 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:25.391768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.391641 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\") pod \"601ada4f-cb28-4958-8ddb-4e8974621dd0\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " Oct 30 06:02:25.392063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.391962 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\" (UniqueName: \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\") pod \"601ada4f-cb28-4958-8ddb-4e8974621dd0\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " Oct 30 06:02:25.392063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.392013 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\") pod \"601ada4f-cb28-4958-8ddb-4e8974621dd0\" (UID: \"601ada4f-cb28-4958-8ddb-4e8974621dd0\") " Oct 30 06:02:25.393005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.392975 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca" pod "601ada4f-cb28-4958-8ddb-4e8974621dd0" (UID: "601ada4f-cb28-4958-8ddb-4e8974621dd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:25.394793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.393222 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:25.395873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.395834 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2" pod "601ada4f-cb28-4958-8ddb-4e8974621dd0" (UID: "601ada4f-cb28-4958-8ddb-4e8974621dd0"). InnerVolumeSpecName "kube-api-access-hjnk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:25.399200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.399166 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca" pod "601ada4f-cb28-4958-8ddb-4e8974621dd0" (UID: "601ada4f-cb28-4958-8ddb-4e8974621dd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:25.495425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.495381 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjnk2\" (UniqueName: \"kubernetes.io/projected/601ada4f-cb28-4958-8ddb-4e8974621dd0-kube-api-access-hjnk2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:25.495425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:25.495424 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/601ada4f-cb28-4958-8ddb-4e8974621dd0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:26.149419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:26.149389 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" Oct 30 06:02:26.149419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:26.149410 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv" event={"ID":"601ada4f-cb28-4958-8ddb-4e8974621dd0","Type":"ContainerDied","Data":"aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639"} Oct 30 06:02:26.149641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:26.149455 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa25ad54f0eb33323762f128bace458a503a44daf8de2104a1a80e38e62a1639" Oct 30 06:02:26.163150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:26.163125 6171 scope.go:122] "RemoveContainer" containerID="f01a80008449db314afcf0b6cfc2d192c45500a953d3c6d38c514cf50d9f457a" Oct 30 06:02:26.382436 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:02:26.382396 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:02:29.279056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.279011 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:02:29.280201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.280179 6171 cpu_manager.go:414] "containerMap: removing container" podUID="601ada4f-cb28-4958-8ddb-4e8974621dd0" containerName="revoke-keys" Oct 30 06:02:29.280201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.280200 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="601ada4f-cb28-4958-8ddb-4e8974621dd0" containerName="revoke-keys" Oct 30 06:02:29.280393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.280379 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="601ada4f-cb28-4958-8ddb-4e8974621dd0" containerName="revoke-keys" Oct 30 06:02:29.286312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.286292 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.293870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.293844 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:02:29.307066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.307026 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.312956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.311619 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:02:29.326872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.326848 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:02:29.467937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.467907 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.467963 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468119 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468160 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468246 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468279 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468304 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468357 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468399 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468426 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2wkd\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468481 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468620 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468650 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468756 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468779 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.468871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468834 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.468969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.468947 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzxr8\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.570131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570034 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570117 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570177 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570202 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570226 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570285 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570310 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.570349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570343 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2wkd\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570405 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570497 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570564 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570594 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.570732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570666 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.571206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570693 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.571206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.570770 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.571336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571301 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.571407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571339 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.571736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571710 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.571877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571851 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.571943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571873 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.571954 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.572113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572091 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzxr8\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572129 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572212 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572352 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.572457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572429 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572640 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572705 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.572945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572921 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.573007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.572928 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.573204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.573189 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.573985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.573962 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.595561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.595538 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.595883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.595863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2wkd\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.596362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.596330 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.597268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.597250 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzxr8\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\") pod \"e2e-inf-a-387b16-openshift-default-5bb4745897-498mp\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.617708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.617674 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:29.633795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.633772 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:29.967905 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:02:29.967868 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678700b0_4562_45e6_ad2a_f987fac4b574.slice/crio-e4d0e7308e293088e75cdb1351f5a8827ac680eb7118c478f69900492c69fec5 WatchSource:0}: Error finding container e4d0e7308e293088e75cdb1351f5a8827ac680eb7118c478f69900492c69fec5: Status 404 returned error can't find the container with id e4d0e7308e293088e75cdb1351f5a8827ac680eb7118c478f69900492c69fec5 Oct 30 06:02:29.970311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.970286 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:02:29.971338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.971303 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:29.971428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.971371 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:29.971428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.971402 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:29.977235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.977211 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:02:29.989745 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:02:29.989717 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac3aa7f6_fbc6_484f_9764_b25f17edb471.slice/crio-8c88efba3b1206e6dc22e91fbe73ff17d4295d894e7329e643dbac51f6b5fc20 WatchSource:0}: Error finding container 8c88efba3b1206e6dc22e91fbe73ff17d4295d894e7329e643dbac51f6b5fc20: Status 404 returned error can't find the container with id 8c88efba3b1206e6dc22e91fbe73ff17d4295d894e7329e643dbac51f6b5fc20 Oct 30 06:02:30.009987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.993667 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:30.009987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.993771 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:30.009987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:29.993830 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:02:30.197664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.197626 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" event={"ID":"678700b0-4562-45e6-ad2a-f987fac4b574","Type":"ContainerStarted","Data":"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4"} Oct 30 06:02:30.197852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.197680 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" event={"ID":"678700b0-4562-45e6-ad2a-f987fac4b574","Type":"ContainerStarted","Data":"e4d0e7308e293088e75cdb1351f5a8827ac680eb7118c478f69900492c69fec5"} Oct 30 06:02:30.199921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.199895 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" event={"ID":"ac3aa7f6-fbc6-484f-9764-b25f17edb471","Type":"ContainerStarted","Data":"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992"} Oct 30 06:02:30.200078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.199935 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" event={"ID":"ac3aa7f6-fbc6-484f-9764-b25f17edb471","Type":"ContainerStarted","Data":"8c88efba3b1206e6dc22e91fbe73ff17d4295d894e7329e643dbac51f6b5fc20"} Oct 30 06:02:30.248145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.248080 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podStartSLOduration=1.248061833 podStartE2EDuration="1.248061833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:30.24660897 +0000 UTC m=+1808.686986166" watchObservedRunningTime="2026-10-30 06:02:30.248061833 +0000 UTC m=+1808.688439027" Oct 30 06:02:30.299481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.299432 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podStartSLOduration=1.2994200949999999 podStartE2EDuration="1.299420095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:30.299322154 +0000 UTC m=+1808.739699349" watchObservedRunningTime="2026-10-30 06:02:30.299420095 +0000 UTC m=+1808.739797287" Oct 30 06:02:30.617852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.617771 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:30.635177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:30.635145 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:31.637116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:31.635446 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:31.637116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:31.635553 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:31.637116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:31.635750 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:31.637116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:31.635836 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:32.637073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:32.624146 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:32.637073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:32.624212 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:32.637073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:32.635884 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:32.637073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:32.635955 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:33.638074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:33.626426 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:33.638074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:33.626502 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:33.641081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:33.640150 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:33.641081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:33.640208 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:34.628070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:34.627624 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:34.628070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:34.627691 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:34.636674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:34.636641 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:34.636831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:34.636705 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:35.625063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:35.622144 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:35.625063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:35.622220 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:35.653067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:35.648069 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:35.653067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:35.648137 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:36.251421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.251377 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" containerID="cri-o://fffde35bd487a21e9cc787707def8e80f23cedfe5364891da52a563f3319e57f" gracePeriod=45 Oct 30 06:02:36.550713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.550475 6171 generic.go:356] "Generic (PLEG): container finished" podID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerID="fffde35bd487a21e9cc787707def8e80f23cedfe5364891da52a563f3319e57f" exitCode=0 Oct 30 06:02:36.550713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.550688 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" event={"ID":"a5f8469a-a47d-4167-9c92-e157b60f4508","Type":"ContainerDied","Data":"fffde35bd487a21e9cc787707def8e80f23cedfe5364891da52a563f3319e57f"} Oct 30 06:02:36.646151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.645955 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" start-of-body= Oct 30 06:02:36.648251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.646025 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" start-of-body= Oct 30 06:02:36.648251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.646120 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.250:15021/healthz/ready\": dial tcp 10.133.0.250:15021: connect: connection refused" Oct 30 06:02:36.648251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.646357 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.249:15021/healthz/ready\": dial tcp 10.133.0.249:15021: connect: connection refused" Oct 30 06:02:36.750730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.750684 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:02:36.863239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.862973 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.863239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863090 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.864074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863280 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.864074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863372 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.864074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863484 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.864074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863506 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\" (UniqueName: \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\") pod \"a5f8469a-a47d-4167-9c92-e157b60f4508\" (UID: \"a5f8469a-a47d-4167-9c92-e157b60f4508\") " Oct 30 06:02:36.864074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.863980 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:36.864388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.864196 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:36.866128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.864537 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:36.866128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.864561 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:36.866128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.864676 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:36.870162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.867362 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:36.870162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.867410 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "kube-api-access-jzk4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:36.870162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.867476 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs" pod "a5f8469a-a47d-4167-9c92-e157b60f4508" (UID: "a5f8469a-a47d-4167-9c92-e157b60f4508"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:36.966486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.966434 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:36.966486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.966487 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a5f8469a-a47d-4167-9c92-e157b60f4508-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:36.966486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.966505 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a5f8469a-a47d-4167-9c92-e157b60f4508-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:36.966851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:36.966519 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzk4n\" (UniqueName: \"kubernetes.io/projected/a5f8469a-a47d-4167-9c92-e157b60f4508-kube-api-access-jzk4n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:37.562784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.562754 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" Oct 30 06:02:37.562983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.562937 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx" event={"ID":"a5f8469a-a47d-4167-9c92-e157b60f4508","Type":"ContainerDied","Data":"3f06cd664fda62dc9454bf3696ef94b20d431eddd63d48fff886accf896bc810"} Oct 30 06:02:37.562983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.562980 6171 scope.go:122] "RemoveContainer" containerID="fffde35bd487a21e9cc787707def8e80f23cedfe5364891da52a563f3319e57f" Oct 30 06:02:37.623901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.623876 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:37.624291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.624268 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:37.625522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.625504 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:02:37.629021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.628999 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 06:02:37.640097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.640039 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:37.640389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.640355 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:37.641138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.641120 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:02:37.642980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.642960 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-862dc2/test-model-862dc2-kserve-dd7448f-qbkjx"] Oct 30 06:02:37.676630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:37.676606 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" path="/var/lib/kubelet/pods/a5f8469a-a47d-4167-9c92-e157b60f4508/volumes" Oct 30 06:02:49.624220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.624186 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 06:02:49.868177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.868110 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb"] Oct 30 06:02:49.877515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.869925 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" Oct 30 06:02:49.882200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.882171 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" Oct 30 06:02:49.882928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.882870 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f8469a-a47d-4167-9c92-e157b60f4508" containerName="main" Oct 30 06:02:49.896353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.894426 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb"] Oct 30 06:02:49.896353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.894552 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:49.919135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.913004 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:02:49.919135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.913170 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:02:49.976638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.976481 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:49.978296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.977419 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:49.978819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:49.978683 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kst6m\" (UniqueName: \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.080959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.080920 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.081176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.080976 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kst6m\" (UniqueName: \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.081269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.081247 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.081695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.081673 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.083705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.083682 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.105984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.105959 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kst6m\" (UniqueName: \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\") pod \"maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.295413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.295377 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:50.610935 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:02:50.610907 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc31b1d4_8715_432d_acc3_25aa3b912cf8.slice/crio-98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112 WatchSource:0}: Error finding container 98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112: Status 404 returned error can't find the container with id 98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112 Oct 30 06:02:50.614136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:50.613676 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb"] Oct 30 06:02:51.021525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:51.021489 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" event={"ID":"fc31b1d4-8715-432d-acc3-25aa3b912cf8","Type":"ContainerStarted","Data":"bb6945bee11572514244c09d01dddea9cb97f080b951a5ba2cb1fb2f3b159cd6"} Oct 30 06:02:51.021878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:51.021548 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" event={"ID":"fc31b1d4-8715-432d-acc3-25aa3b912cf8","Type":"ContainerStarted","Data":"98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112"} Oct 30 06:02:51.067269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:51.067227 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" podStartSLOduration=2.067214304 podStartE2EDuration="2.067214304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:51.065346961 +0000 UTC m=+1829.505724153" watchObservedRunningTime="2026-10-30 06:02:51.067214304 +0000 UTC m=+1829.507591497" Oct 30 06:02:52.082986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:52.082947 6171 generic.go:356] "Generic (PLEG): container finished" podID="fc31b1d4-8715-432d-acc3-25aa3b912cf8" containerID="bb6945bee11572514244c09d01dddea9cb97f080b951a5ba2cb1fb2f3b159cd6" exitCode=0 Oct 30 06:02:52.083726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:52.083698 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" event={"ID":"fc31b1d4-8715-432d-acc3-25aa3b912cf8","Type":"ContainerDied","Data":"bb6945bee11572514244c09d01dddea9cb97f080b951a5ba2cb1fb2f3b159cd6"} Oct 30 06:02:53.373275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.373251 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:53.529663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.529626 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\") pod \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " Oct 30 06:02:53.529958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.529938 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\") pod \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " Oct 30 06:02:53.530090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.529938 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca" pod "fc31b1d4-8715-432d-acc3-25aa3b912cf8" (UID: "fc31b1d4-8715-432d-acc3-25aa3b912cf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:53.530171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.530150 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\" (UniqueName: \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\") pod \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\" (UID: \"fc31b1d4-8715-432d-acc3-25aa3b912cf8\") " Oct 30 06:02:53.530802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.530780 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:53.532124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.532100 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca" pod "fc31b1d4-8715-432d-acc3-25aa3b912cf8" (UID: "fc31b1d4-8715-432d-acc3-25aa3b912cf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:53.532319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.532300 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m" pod "fc31b1d4-8715-432d-acc3-25aa3b912cf8" (UID: "fc31b1d4-8715-432d-acc3-25aa3b912cf8"). InnerVolumeSpecName "kube-api-access-kst6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:53.632639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.632582 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kst6m\" (UniqueName: \"kubernetes.io/projected/fc31b1d4-8715-432d-acc3-25aa3b912cf8-kube-api-access-kst6m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:53.632639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:53.632609 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc31b1d4-8715-432d-acc3-25aa3b912cf8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:02:54.107923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:54.107885 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" Oct 30 06:02:54.108134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:54.107920 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb" event={"ID":"fc31b1d4-8715-432d-acc3-25aa3b912cf8","Type":"ContainerDied","Data":"98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112"} Oct 30 06:02:54.108134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:02:54.107981 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98ea0810b568160661a7c65125e37ce89a2c25cc0cb4bc01d35e21e6238dd112" Oct 30 06:02:56.416880 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:02:56.416840 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:03:02.792147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:02.792106 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 06:03:02.792685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:02.792371 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" podUID="eb0da49d-891d-4c77-baf6-e4596cd2071d" containerName="maas-api" containerID="cri-o://74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8" gracePeriod=30 Oct 30 06:03:03.096734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.096708 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 06:03:03.187067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.184578 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\") pod \"eb0da49d-891d-4c77-baf6-e4596cd2071d\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " Oct 30 06:03:03.187067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.184808 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\") pod \"eb0da49d-891d-4c77-baf6-e4596cd2071d\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " Oct 30 06:03:03.187067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.184901 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\") pod \"eb0da49d-891d-4c77-baf6-e4596cd2071d\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " Oct 30 06:03:03.187067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.185087 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\") pod \"eb0da49d-891d-4c77-baf6-e4596cd2071d\" (UID: \"eb0da49d-891d-4c77-baf6-e4596cd2071d\") " Oct 30 06:03:03.211095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.202002 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca" pod "eb0da49d-891d-4c77-baf6-e4596cd2071d" (UID: "eb0da49d-891d-4c77-baf6-e4596cd2071d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:03.235065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.231477 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls" pod "eb0da49d-891d-4c77-baf6-e4596cd2071d" (UID: "eb0da49d-891d-4c77-baf6-e4596cd2071d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:03.248189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.235823 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6" pod "eb0da49d-891d-4c77-baf6-e4596cd2071d" (UID: "eb0da49d-891d-4c77-baf6-e4596cd2071d"). InnerVolumeSpecName "kube-api-access-g8dn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:03.248189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.235942 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs" pod "eb0da49d-891d-4c77-baf6-e4596cd2071d" (UID: "eb0da49d-891d-4c77-baf6-e4596cd2071d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:03.325126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.292003 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:03.325126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.292061 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8dn6\" (UniqueName: \"kubernetes.io/projected/eb0da49d-891d-4c77-baf6-e4596cd2071d-kube-api-access-g8dn6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:03.325126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.292079 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb0da49d-891d-4c77-baf6-e4596cd2071d-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:03.325126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.292092 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb0da49d-891d-4c77-baf6-e4596cd2071d-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:03.338072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.327767 6171 generic.go:356] "Generic (PLEG): container finished" podID="eb0da49d-891d-4c77-baf6-e4596cd2071d" containerID="74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8" exitCode=0 Oct 30 06:03:03.338072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.328077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" event={"ID":"eb0da49d-891d-4c77-baf6-e4596cd2071d","Type":"ContainerDied","Data":"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8"} Oct 30 06:03:03.338072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.328136 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" event={"ID":"eb0da49d-891d-4c77-baf6-e4596cd2071d","Type":"ContainerDied","Data":"aa070dd2dbaa492e74478e16b889b8f6067997f5727c8c54140375402358fc13"} Oct 30 06:03:03.338072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.328182 6171 scope.go:122] "RemoveContainer" containerID="74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8" Oct 30 06:03:03.338072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.328367 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4" Oct 30 06:03:03.395145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.388732 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410589 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fc31b1d4-8715-432d-acc3-25aa3b912cf8" containerName="revoke-keys" Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410619 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc31b1d4-8715-432d-acc3-25aa3b912cf8" containerName="revoke-keys" Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410659 6171 cpu_manager.go:414] "containerMap: removing container" podUID="eb0da49d-891d-4c77-baf6-e4596cd2071d" containerName="maas-api" Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410667 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb0da49d-891d-4c77-baf6-e4596cd2071d" containerName="maas-api" Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410952 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc31b1d4-8715-432d-acc3-25aa3b912cf8" containerName="revoke-keys" Oct 30 06:03:03.424074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.410976 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb0da49d-891d-4c77-baf6-e4596cd2071d" containerName="maas-api" Oct 30 06:03:03.446073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.439428 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.452069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.446414 6171 scope.go:122] "RemoveContainer" containerID="74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8" Oct 30 06:03:03.484064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.481308 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:03:03.492066 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:03.484653 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8\": container with ID starting with 74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8 not found: ID does not exist" containerID="74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8" Oct 30 06:03:03.492066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.484756 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8"} err="failed to get container status \"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8\": rpc error: code = NotFound desc = could not find container \"74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8\": container with ID starting with 74aca0c94a64e526c5120916a13f1078be25f5ea34451fe4c825e6bfba38a4c8 not found: ID does not exist" Oct 30 06:03:03.560070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.557969 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-c54211-serving-cert\"" Oct 30 06:03:03.632120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.631983 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.632407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.632155 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.632407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.632291 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.632407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.632320 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2xmr\" (UniqueName: \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.634170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.634024 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 06:03:03.638973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.638936 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-84dc9d-7cf58856b8-7tfk4"] Oct 30 06:03:03.679330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.679289 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb0da49d-891d-4c77-baf6-e4596cd2071d" path="/var/lib/kubelet/pods/eb0da49d-891d-4c77-baf6-e4596cd2071d/volumes" Oct 30 06:03:03.734463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.734435 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.734635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.734468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2xmr\" (UniqueName: \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.734635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.734630 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.734756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.734734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.735353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.735328 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.737190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.737172 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.737299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.737281 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:03.815634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:03.815568 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2xmr\" (UniqueName: \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\") pod \"maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:04.109055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:04.109021 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:04.426397 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:04.426365 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod405e2758_598d_40d8_b3e0_7ac155cbe01f.slice/crio-47564442e58cfb5392cde4e88d63156a846e4af5448554c35744a127b21df649 WatchSource:0}: Error finding container 47564442e58cfb5392cde4e88d63156a846e4af5448554c35744a127b21df649: Status 404 returned error can't find the container with id 47564442e58cfb5392cde4e88d63156a846e4af5448554c35744a127b21df649 Oct 30 06:03:04.429228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:04.429207 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:03:04.650442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:04.650318 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" containerID="cri-o://b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26" gracePeriod=45 Oct 30 06:03:04.776498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:04.776458 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" probeResult="failure" output="Get \"https://10.133.0.213:8000/ready\": dial tcp 10.133.0.213:8000: connect: connection refused" Oct 30 06:03:04.986291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:04.986257 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:03:05.080388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080356 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.080602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080442 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.080602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080490 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.080602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080521 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.080792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080623 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\" (UniqueName: \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.080792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.080684 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\" (UniqueName: \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\") pod \"7264c9e4-2f72-4cac-b559-cdc96c372d24\" (UID: \"7264c9e4-2f72-4cac-b559-cdc96c372d24\") " Oct 30 06:03:05.081554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.081120 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:05.081554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.081254 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:05.081554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.081507 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.081554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.081528 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.081903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.081885 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:05.083487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.083459 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:05.083785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.083765 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:05.083930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.083911 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2" pod "7264c9e4-2f72-4cac-b559-cdc96c372d24" (UID: "7264c9e4-2f72-4cac-b559-cdc96c372d24"). InnerVolumeSpecName "kube-api-access-6lfb2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:05.183666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.183633 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.183666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.183666 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7264c9e4-2f72-4cac-b559-cdc96c372d24-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.183911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.183682 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfb2\" (UniqueName: \"kubernetes.io/projected/7264c9e4-2f72-4cac-b559-cdc96c372d24-kube-api-access-6lfb2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.183911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.183693 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7264c9e4-2f72-4cac-b559-cdc96c372d24-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:05.350973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.350941 6171 generic.go:356] "Generic (PLEG): container finished" podID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerID="b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26" exitCode=0 Oct 30 06:03:05.351202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.350992 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" Oct 30 06:03:05.351202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.351029 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" event={"ID":"7264c9e4-2f72-4cac-b559-cdc96c372d24","Type":"ContainerDied","Data":"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26"} Oct 30 06:03:05.351202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.351107 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f" event={"ID":"7264c9e4-2f72-4cac-b559-cdc96c372d24","Type":"ContainerDied","Data":"3cf1ac49d22027817802c99876db53de33f78b2859dbbe3d51db9f42ab00c562"} Oct 30 06:03:05.351202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.351126 6171 scope.go:122] "RemoveContainer" containerID="b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26" Oct 30 06:03:05.353015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.352993 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" event={"ID":"405e2758-598d-40d8-b3e0-7ac155cbe01f","Type":"ContainerStarted","Data":"4325a2de7afd7d03c1147c4b8f21b0758e37a27496570c7920cd703405010f38"} Oct 30 06:03:05.353207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.353055 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" event={"ID":"405e2758-598d-40d8-b3e0-7ac155cbe01f","Type":"ContainerStarted","Data":"47564442e58cfb5392cde4e88d63156a846e4af5448554c35744a127b21df649"} Oct 30 06:03:05.353207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.353133 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:05.365234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.365215 6171 scope.go:122] "RemoveContainer" containerID="b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26" Oct 30 06:03:05.365489 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:05.365466 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26\": container with ID starting with b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26 not found: ID does not exist" containerID="b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26" Oct 30 06:03:05.365546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.365501 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26"} err="failed to get container status \"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26\": rpc error: code = NotFound desc = could not find container \"b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26\": container with ID starting with b035f39a5b457fafeacffa69431d70bed654f225c57069581365b9cdfc34be26 not found: ID does not exist" Oct 30 06:03:05.417052 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.417009 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" podStartSLOduration=34.064792521 podStartE2EDuration="34.416997753s" totalImagesPullingTime="352.205232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:05.413736113 +0000 UTC m=+1843.854113307" watchObservedRunningTime="2026-10-30 06:03:05.416997753 +0000 UTC m=+1843.857374953" Oct 30 06:03:05.459229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.459204 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 06:03:05.464259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.464240 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-de7242/test-model-de7242-kserve-c9bf79595-bfl2f"] Oct 30 06:03:05.713170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:05.713089 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" path="/var/lib/kubelet/pods/7264c9e4-2f72-4cac-b559-cdc96c372d24/volumes" Oct 30 06:03:11.369898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:11.369866 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:03:11.442893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:11.442862 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 06:03:11.443216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:11.443086 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" containerID="cri-o://18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8" gracePeriod=30 Oct 30 06:03:12.009613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.009576 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x"] Oct 30 06:03:12.010579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.010562 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" Oct 30 06:03:12.010657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.010581 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" Oct 30 06:03:12.010812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.010801 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7264c9e4-2f72-4cac-b559-cdc96c372d24" containerName="main" Oct 30 06:03:12.016400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.016375 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.036470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.036446 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x"] Oct 30 06:03:12.143198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.124286 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.177903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.145308 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:03:12.177903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.145373 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:03:12.194618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.178365 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdjrj\" (UniqueName: \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.194618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.191878 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.293542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.293450 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.293722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.293568 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdjrj\" (UniqueName: \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.293722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.293704 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.294386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.294351 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.296094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.296061 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.318876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.318850 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdjrj\" (UniqueName: \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\") pod \"maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.331725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.331701 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:12.639932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.639852 6171 patch_prober.go:29] interesting pod/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:03:12.639932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.639901 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:03:12.658815 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:12.658786 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf788b086_a6f9_4955_aa30_36cf491a9eda.slice/crio-06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4 WatchSource:0}: Error finding container 06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4: Status 404 returned error can't find the container with id 06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4 Oct 30 06:03:12.679908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.679817 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x"] Oct 30 06:03:12.734322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:12.734291 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" event={"ID":"f788b086-a6f9-4955-aa30-36cf491a9eda","Type":"ContainerStarted","Data":"06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4"} Oct 30 06:03:13.745438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:13.745403 6171 generic.go:356] "Generic (PLEG): container finished" podID="f788b086-a6f9-4955-aa30-36cf491a9eda" containerID="c50b16cd379bf980ff93cbd7e202fe48abd3bdd0af70592145b4186bd13dd717" exitCode=0 Oct 30 06:03:13.745777 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:13.745479 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" event={"ID":"f788b086-a6f9-4955-aa30-36cf491a9eda","Type":"ContainerDied","Data":"c50b16cd379bf980ff93cbd7e202fe48abd3bdd0af70592145b4186bd13dd717"} Oct 30 06:03:14.961172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:14.961145 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:15.058112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.058015 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\") pod \"f788b086-a6f9-4955-aa30-36cf491a9eda\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " Oct 30 06:03:15.058112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.058086 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\") pod \"f788b086-a6f9-4955-aa30-36cf491a9eda\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " Oct 30 06:03:15.058350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.058199 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\" (UniqueName: \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\") pod \"f788b086-a6f9-4955-aa30-36cf491a9eda\" (UID: \"f788b086-a6f9-4955-aa30-36cf491a9eda\") " Oct 30 06:03:15.058416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.058388 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca" pod "f788b086-a6f9-4955-aa30-36cf491a9eda" (UID: "f788b086-a6f9-4955-aa30-36cf491a9eda"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:15.059300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.059271 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:15.060489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.060462 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca" pod "f788b086-a6f9-4955-aa30-36cf491a9eda" (UID: "f788b086-a6f9-4955-aa30-36cf491a9eda"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:15.060602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.060561 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj" pod "f788b086-a6f9-4955-aa30-36cf491a9eda" (UID: "f788b086-a6f9-4955-aa30-36cf491a9eda"). InnerVolumeSpecName "kube-api-access-rdjrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:15.161113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.161078 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f788b086-a6f9-4955-aa30-36cf491a9eda-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:15.161113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.161103 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdjrj\" (UniqueName: \"kubernetes.io/projected/f788b086-a6f9-4955-aa30-36cf491a9eda-kube-api-access-rdjrj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:15.225334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.225299 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:03:15.771007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.770979 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" Oct 30 06:03:15.771211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.771009 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x" event={"ID":"f788b086-a6f9-4955-aa30-36cf491a9eda","Type":"ContainerDied","Data":"06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4"} Oct 30 06:03:15.771211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:15.771102 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06103cc49d1e1dd7e1f47edd7ed04031cf8fdbd7583a86e3f7ba0bf20f1331a4" Oct 30 06:03:16.933197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:16.929256 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111340 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111472 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111559 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111597 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111717 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111835 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111890 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111962 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.112124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.111994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\") pod \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\" (UID: \"cc8eba30-ea37-4750-939f-dd56ff6d6e9e\") " Oct 30 06:03:17.118471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.118420 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:17.122276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.122245 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:17.124414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.124374 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:17.125579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.125149 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:17.225147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.197709 6171 generic.go:356] "Generic (PLEG): container finished" podID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerID="18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8" exitCode=0 Oct 30 06:03:17.225147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.197898 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerDied","Data":"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8"} Oct 30 06:03:17.225147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.198002 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" event={"ID":"cc8eba30-ea37-4750-939f-dd56ff6d6e9e","Type":"ContainerDied","Data":"8891436393b19288f202b042422f1ca14ccee79e2c41c91d4a9ca096ad6de79b"} Oct 30 06:03:17.225958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.225906 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "kube-api-access-khqm6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:17.253083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.233100 6171 scope.go:122] "RemoveContainer" containerID="18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8" Oct 30 06:03:17.253083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.233446 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.265397 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273449 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273476 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273493 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273513 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273545 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khqm6\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-kube-api-access-khqm6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.275082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.273562 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.349080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.336285 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:17.349558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.349521 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:17.414155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.384104 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.453069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.424303 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.530891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.530845 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token" pod "cc8eba30-ea37-4750-939f-dd56ff6d6e9e" (UID: "cc8eba30-ea37-4750-939f-dd56ff6d6e9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:17.533122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.533097 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc8eba30-ea37-4750-939f-dd56ff6d6e9e-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:17.937076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:17.895964 6171 scope.go:122] "RemoveContainer" containerID="5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449" Oct 30 06:03:18.175089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.172616 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:18.256072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.250180 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:03:18.256072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.251653 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f788b086-a6f9-4955-aa30-36cf491a9eda" containerName="revoke-keys" Oct 30 06:03:18.256072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.251680 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f788b086-a6f9-4955-aa30-36cf491a9eda" containerName="revoke-keys" Oct 30 06:03:18.256072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.251764 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.256072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.251771 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.303072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.293234 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f788b086-a6f9-4955-aa30-36cf491a9eda" containerName="revoke-keys" Oct 30 06:03:18.303072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.293297 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.303072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.293327 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.311066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.309322 6171 scope.go:122] "RemoveContainer" containerID="18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8" Oct 30 06:03:18.315064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.311424 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:03:18.325153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.322158 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:18.351751 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:18.329446 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8\": container with ID starting with 18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8 not found: ID does not exist" containerID="18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8" Oct 30 06:03:18.351751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.329495 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8"} err="failed to get container status \"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8\": rpc error: code = NotFound desc = could not find container \"18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8\": container with ID starting with 18ea86366e5e39de041e77cc78831c5c0304ca5a7f4128cbaa99673fea6444e8 not found: ID does not exist" Oct 30 06:03:18.351751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.329543 6171 scope.go:122] "RemoveContainer" containerID="5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449" Oct 30 06:03:18.351751 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:18.349299 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449\": container with ID starting with 5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449 not found: ID does not exist" containerID="5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449" Oct 30 06:03:18.351751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.349342 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449"} err="failed to get container status \"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449\": rpc error: code = NotFound desc = could not find container \"5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449\": container with ID starting with 5a2f6b5c4c954835ec74d9c176e941b7d9260079b586b6ece420f377c2545449 not found: ID does not exist" Oct 30 06:03:18.370242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.367333 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.398146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.386083 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:03:18.398146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.395841 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d64c9-openshift-default-dockercfg-zxnwz\"" Oct 30 06:03:18.398146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.395885 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 06:03:18.403705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.403670 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-84dc9d-openshift-default-54749c6469-cpr62"] Oct 30 06:03:18.420400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.420363 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.420792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.420774 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.420942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.420927 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.421210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.421193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.421367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.421344 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.421478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.421463 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqbs9\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.421620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.421602 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.421927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.421903 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.422209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.422190 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.523620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523583 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.523856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523643 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.523856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523762 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.523856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523795 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523904 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523933 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.523974 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqbs9\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524020 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524170 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524235 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524597 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.524960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524940 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.525096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.524989 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.525300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.525176 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.526944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.526920 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.527244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.527223 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.549955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.549926 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.551101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.551072 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqbs9\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\") pod \"e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.694198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.693789 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:03:18.721346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.695453 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.721346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.695480 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" containerName="istio-proxy" Oct 30 06:03:18.721346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.702126 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.721346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.720507 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4e35c\"/\"default-dockercfg-l8wnp\"" Oct 30 06:03:18.721346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.721102 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:18.728163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728121 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.728296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728177 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.728384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728291 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrt5c\" (UniqueName: \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.728384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728365 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.728550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728425 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.728550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.728521 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.731034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.730912 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:03:18.736953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.736934 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4e35c\"/\"openshift-service-ca.crt\"" Oct 30 06:03:18.737083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.737037 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f4e35c\"/\"kube-root-ca.crt\"" Oct 30 06:03:18.812070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.811317 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f4e35c\"/\"test-model-f4e35c-kserve-self-signed-certs\"" Oct 30 06:03:18.833657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.833633 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.833782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.833683 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrt5c\" (UniqueName: \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.833782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.833731 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.833782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.833774 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.833970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.833802 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.834082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.834070 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.834167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.834133 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.834385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.834335 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.834682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.834661 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.836932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.836661 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.837263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.837244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:18.859399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:18.859374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrt5c\" (UniqueName: \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\") pod \"test-model-f4e35c-kserve-75bfd44d7-lgzh7\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:19.089179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.088703 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:19.211645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.181893 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:03:19.378776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.377908 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:19.378776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.378008 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:19.378776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.378073 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:19.963691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.963656 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc8eba30-ea37-4750-939f-dd56ff6d6e9e" path="/var/lib/kubelet/pods/cc8eba30-ea37-4750-939f-dd56ff6d6e9e/volumes" Oct 30 06:03:19.964660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:19.964630 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerStarted","Data":"7eacb3e44ad1e1ae6c27854086c2e7a6e8f311d7c0173eddc383d2cd56aaadd3"} Oct 30 06:03:20.348066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:20.347932 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:03:20.356730 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:20.356701 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd0471402_79d6_4cae_b25e_1a1745e1f69f.slice/crio-43a35d771cec608e27447a5dcdd9948c5cbf31b80bb00daf1dd6afc31ab2ff4b WatchSource:0}: Error finding container 43a35d771cec608e27447a5dcdd9948c5cbf31b80bb00daf1dd6afc31ab2ff4b: Status 404 returned error can't find the container with id 43a35d771cec608e27447a5dcdd9948c5cbf31b80bb00daf1dd6afc31ab2ff4b Oct 30 06:03:20.969092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:20.969020 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerStarted","Data":"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153"} Oct 30 06:03:20.970615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:20.970591 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" event={"ID":"d0471402-79d6-4cae-b25e-1a1745e1f69f","Type":"ContainerStarted","Data":"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d"} Oct 30 06:03:20.970775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:20.970764 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" event={"ID":"d0471402-79d6-4cae-b25e-1a1745e1f69f","Type":"ContainerStarted","Data":"43a35d771cec608e27447a5dcdd9948c5cbf31b80bb00daf1dd6afc31ab2ff4b"} Oct 30 06:03:21.046270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:21.046222 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podStartSLOduration=30.046206275 podStartE2EDuration="30.046206275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:21.030149418 +0000 UTC m=+1859.470526616" watchObservedRunningTime="2026-10-30 06:03:21.046206275 +0000 UTC m=+1859.486583472" Oct 30 06:03:21.078882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:21.078835 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podStartSLOduration=3.078822829 podStartE2EDuration="3.078822829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:21.074676146 +0000 UTC m=+1859.515053374" watchObservedRunningTime="2026-10-30 06:03:21.078822829 +0000 UTC m=+1859.519200022" Oct 30 06:03:21.104069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:21.103729 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:21.741363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:21.740561 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:21.747029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:21.747005 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:22.025121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:22.016715 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:22.030070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:22.029229 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:03:22.695065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:22.694092 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:03:22.769510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:22.769096 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:03:25.444750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.444715 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:03:25.451487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.451466 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.475156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.475126 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:03:25.504070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.480131 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-0d64c9\"" Oct 30 06:03:25.523676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.523644 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:03:25.527804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.527779 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.528142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.528119 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzvh6\" (UniqueName: \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.530069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.530054 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:25.546216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.546190 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:03:25.549448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.549426 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-0d64c9-dockercfg-mrqwd\"" Oct 30 06:03:25.781076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.780272 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzvh6\" (UniqueName: \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.781076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.780494 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.781076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.780588 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:25.781076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.780639 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlp6r\" (UniqueName: \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:25.812478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.782862 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:25.921276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.921240 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:25.921560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.921539 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlp6r\" (UniqueName: \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:25.923871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:25.923844 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:26.041139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.040992 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlp6r\" (UniqueName: \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\") pod \"payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:26.042721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.042691 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzvh6\" (UniqueName: \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\") pod \"payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:26.072730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.072700 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:26.164323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.164290 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:26.224549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.224011 6171 scope.go:122] "RemoveContainer" containerID="2100670daaaf02e22db323cc82687b07c604f8eb7d804dc91602c4c9bff03535" Oct 30 06:03:26.285105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.284650 6171 scope.go:122] "RemoveContainer" containerID="9fcd70c13398bc400a7fa40576333d0979f50609f721573cfcd8b231c34c40e1" Oct 30 06:03:26.402838 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:26.402798 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:03:26.413672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.413356 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:03:26.519773 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:26.519743 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213bc107_2f1f_4a90_bd30_29ed691231c8.slice/crio-bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9 WatchSource:0}: Error finding container bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9: Status 404 returned error can't find the container with id bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9 Oct 30 06:03:26.521816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:26.521786 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:03:27.307217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.307165 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" event={"ID":"213bc107-2f1f-4a90-bd30-29ed691231c8","Type":"ContainerStarted","Data":"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b"} Oct 30 06:03:27.307417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.307245 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" event={"ID":"213bc107-2f1f-4a90-bd30-29ed691231c8","Type":"ContainerStarted","Data":"bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9"} Oct 30 06:03:27.307417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.307313 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:27.311178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.311139 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" event={"ID":"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d","Type":"ContainerStarted","Data":"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e"} Oct 30 06:03:27.311340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.311195 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" event={"ID":"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d","Type":"ContainerStarted","Data":"868c3385d9d33495c469aabc430b20e8a48dbb2bf702456ab7192a887029a84b"} Oct 30 06:03:27.311340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.311235 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:27.355775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.355725 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" podStartSLOduration=2.355711865 podStartE2EDuration="2.355711865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:27.352275035 +0000 UTC m=+1865.792652229" watchObservedRunningTime="2026-10-30 06:03:27.355711865 +0000 UTC m=+1865.796089058" Oct 30 06:03:27.400738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.400676 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" podStartSLOduration=2.400657287 podStartE2EDuration="2.400657287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:27.396889788 +0000 UTC m=+1865.837266980" watchObservedRunningTime="2026-10-30 06:03:27.400657287 +0000 UTC m=+1865.841034484" Oct 30 06:03:27.694963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.694831 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" podUID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" containerName="payload-processing" containerID="cri-o://4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb" gracePeriod=25 Oct 30 06:03:27.770173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:27.770094 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" podUID="38c144f1-2255-40ea-926f-01b4bd35feea" containerName="payload-pre-processing" containerID="cri-o://f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db" gracePeriod=25 Oct 30 06:03:28.166127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.166094 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:03:28.170217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.170185 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:03:28.261872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.261835 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\" (UniqueName: \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\") pod \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " Oct 30 06:03:28.262112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.261912 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\" (UniqueName: \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\") pod \"38c144f1-2255-40ea-926f-01b4bd35feea\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " Oct 30 06:03:28.262112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.261978 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\") pod \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\" (UID: \"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc\") " Oct 30 06:03:28.262338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.262308 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\") pod \"38c144f1-2255-40ea-926f-01b4bd35feea\" (UID: \"38c144f1-2255-40ea-926f-01b4bd35feea\") " Oct 30 06:03:28.264004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.263974 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume" pod "0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" (UID: "0aa73dc7-348b-4200-b7b5-d2eb647fcfbc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:28.264292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.264167 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume" pod "38c144f1-2255-40ea-926f-01b4bd35feea" (UID: "38c144f1-2255-40ea-926f-01b4bd35feea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:28.267225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.267192 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv" pod "38c144f1-2255-40ea-926f-01b4bd35feea" (UID: "38c144f1-2255-40ea-926f-01b4bd35feea"). InnerVolumeSpecName "kube-api-access-7fmfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:28.267988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.267934 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx" pod "0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" (UID: "0aa73dc7-348b-4200-b7b5-d2eb647fcfbc"). InnerVolumeSpecName "kube-api-access-lqcfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:28.328194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.328154 6171 generic.go:356] "Generic (PLEG): container finished" podID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" containerID="4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb" exitCode=0 Oct 30 06:03:28.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.328202 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" Oct 30 06:03:28.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.328250 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" event={"ID":"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc","Type":"ContainerDied","Data":"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb"} Oct 30 06:03:28.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.328309 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5" event={"ID":"0aa73dc7-348b-4200-b7b5-d2eb647fcfbc","Type":"ContainerDied","Data":"a35cc964e026dc83d1d69ff823f467f582057fc40cbedb04c2108b681237dd21"} Oct 30 06:03:28.328409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.328333 6171 scope.go:122] "RemoveContainer" containerID="4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb" Oct 30 06:03:28.334428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.334396 6171 generic.go:356] "Generic (PLEG): container finished" podID="38c144f1-2255-40ea-926f-01b4bd35feea" containerID="f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db" exitCode=0 Oct 30 06:03:28.334577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.334462 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" Oct 30 06:03:28.334796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.334670 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" event={"ID":"38c144f1-2255-40ea-926f-01b4bd35feea","Type":"ContainerDied","Data":"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db"} Oct 30 06:03:28.334796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.334727 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52" event={"ID":"38c144f1-2255-40ea-926f-01b4bd35feea","Type":"ContainerDied","Data":"59a47ffa8761b8965015b15a506445f44acb0d03b499ae33e939239fff5534b4"} Oct 30 06:03:28.348849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.348823 6171 scope.go:122] "RemoveContainer" containerID="4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb" Oct 30 06:03:28.349269 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:28.349245 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb\": container with ID starting with 4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb not found: ID does not exist" containerID="4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb" Oct 30 06:03:28.349359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.349278 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb"} err="failed to get container status \"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb\": rpc error: code = NotFound desc = could not find container \"4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb\": container with ID starting with 4904465f5146c8ccbe7b072ced40d3409549a440d9add9bb24c257340fcfa0eb not found: ID does not exist" Oct 30 06:03:28.349359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.349303 6171 scope.go:122] "RemoveContainer" containerID="f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db" Oct 30 06:03:28.363702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.363674 6171 scope.go:122] "RemoveContainer" containerID="f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db" Oct 30 06:03:28.364158 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:28.364129 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db\": container with ID starting with f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db not found: ID does not exist" containerID="f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db" Oct 30 06:03:28.364275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.364169 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db"} err="failed to get container status \"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db\": rpc error: code = NotFound desc = could not find container \"f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db\": container with ID starting with f86b7c6ec7a3bbbd62c7bb30655ce71d1796304b2fe0f87ad4b4bbd1659652db not found: ID does not exist" Oct 30 06:03:28.365320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.365298 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38c144f1-2255-40ea-926f-01b4bd35feea-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:28.365399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.365323 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqcfx\" (UniqueName: \"kubernetes.io/projected/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-kube-api-access-lqcfx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:28.365399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.365333 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmfv\" (UniqueName: \"kubernetes.io/projected/38c144f1-2255-40ea-926f-01b4bd35feea-kube-api-access-7fmfv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:28.365399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.365345 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:28.407491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.407449 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:03:28.413105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.413069 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-f78ca5-78696c64f-5sss5"] Oct 30 06:03:28.448677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.448644 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:03:28.456541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:28.456506 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-f78ca5-679f7559f-cnl52"] Oct 30 06:03:29.101979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.101080 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:29.121449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.120997 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:29.340226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.340184 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:03:29.340444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.340248 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:29.342209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.342176 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:03:29.342365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.342267 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:29.367003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.366919 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:03:29.682193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.682113 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" path="/var/lib/kubelet/pods/0aa73dc7-348b-4200-b7b5-d2eb647fcfbc/volumes" Oct 30 06:03:29.682643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:29.682621 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38c144f1-2255-40ea-926f-01b4bd35feea" path="/var/lib/kubelet/pods/38c144f1-2255-40ea-926f-01b4bd35feea/volumes" Oct 30 06:03:32.409590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.409558 6171 generic.go:356] "Generic (PLEG): container finished" podID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerID="74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5" exitCode=0 Oct 30 06:03:32.410115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.409627 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerDied","Data":"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5"} Oct 30 06:03:32.410115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.410105 6171 scope.go:122] "RemoveContainer" containerID="74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5" Oct 30 06:03:32.411887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.411863 6171 generic.go:356] "Generic (PLEG): container finished" podID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerID="ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477" exitCode=0 Oct 30 06:03:32.412015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.411940 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerDied","Data":"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477"} Oct 30 06:03:32.412215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.412189 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:32.412276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.412264 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:32.412325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.412312 6171 scope.go:122] "RemoveContainer" containerID="ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477" Oct 30 06:03:32.412373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.412314 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:32.413932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.413903 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:32.414024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.413971 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:32.414024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:32.414005 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:03:33.443156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:33.443114 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerStarted","Data":"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2"} Oct 30 06:03:33.445527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:33.445502 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerStarted","Data":"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe"} Oct 30 06:03:33.847834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:33.847792 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:03:33.848028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:33.847941 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:33.848028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:33.847973 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:34.173664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:34.171825 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:03:34.173664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:34.171887 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:34.173664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:34.171935 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:34.847632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:34.847603 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:34.848061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:34.847663 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:35.172748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.172661 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:35.172748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.172718 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:35.363782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.362686 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364278 6171 cpu_manager.go:414] "containerMap: removing container" podUID="38c144f1-2255-40ea-926f-01b4bd35feea" containerName="payload-pre-processing" Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364307 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38c144f1-2255-40ea-926f-01b4bd35feea" containerName="payload-pre-processing" Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364348 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" containerName="payload-processing" Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364357 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" containerName="payload-processing" Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364630 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="38c144f1-2255-40ea-926f-01b4bd35feea" containerName="payload-pre-processing" Oct 30 06:03:35.365193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.364652 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aa73dc7-348b-4200-b7b5-d2eb647fcfbc" containerName="payload-processing" Oct 30 06:03:35.369501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.369479 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.371961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.371332 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:03:35.419353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.419315 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3cc570\"/\"test-model-3cc570-kserve-self-signed-certs\"" Oct 30 06:03:35.439268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.419683 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-3cc570\"/\"default-dockercfg-jpc8x\"" Oct 30 06:03:35.439268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.419725 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3cc570\"/\"openshift-service-ca.crt\"" Oct 30 06:03:35.439268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.419819 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-3cc570\"/\"kube-root-ca.crt\"" Oct 30 06:03:35.512070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.490622 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.522305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.522268 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.522636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.522613 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.522782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.522766 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.522905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.522889 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.523413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.523375 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4t5d\" (UniqueName: \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630404 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630525 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630635 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630658 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.631069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.630849 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4t5d\" (UniqueName: \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.641685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.641651 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.670069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.650409 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.670069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.650673 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.678370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.678339 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.710111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.703005 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.758644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.758610 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4t5d\" (UniqueName: \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\") pod \"test-model-3cc570-kserve-79d9685cc-jckvf\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:35.851071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.848859 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:35.851071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:35.848920 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:36.030981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.030948 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:36.172935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.172191 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:36.172935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.172247 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:36.449213 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:36.449178 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaf19d653_5b0a_4e26_9609_31d9065f59cc.slice/crio-13b43c0d6d99dd277f6d63208e8d035f10a3b0a59ee815b5a2525c050bc48918 WatchSource:0}: Error finding container 13b43c0d6d99dd277f6d63208e8d035f10a3b0a59ee815b5a2525c050bc48918: Status 404 returned error can't find the container with id 13b43c0d6d99dd277f6d63208e8d035f10a3b0a59ee815b5a2525c050bc48918 Oct 30 06:03:36.455949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.455061 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:03:36.528071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.527105 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" event={"ID":"af19d653-5b0a-4e26-9609-31d9065f59cc","Type":"ContainerStarted","Data":"13b43c0d6d99dd277f6d63208e8d035f10a3b0a59ee815b5a2525c050bc48918"} Oct 30 06:03:36.848343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.848301 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:36.848510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.848363 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:36.960925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.960892 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 06:03:36.961449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:36.961120 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" podUID="22dec94d-c650-478e-b581-63ae985bca08" containerName="maas-api" containerID="cri-o://8024a4e23239859786c996d82356b21ba39098ac3fb3891d831fcff44513e185" gracePeriod=30 Oct 30 06:03:37.173071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:37.172590 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:37.173071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:37.172651 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:37.924135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:37.919680 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:37.924135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:37.919741 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:38.289002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.288966 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:38.309220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.309183 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:38.346071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.316573 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:38.415069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.355525 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:03:38.426379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.367149 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb"] Oct 30 06:03:38.426379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.367430 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" podUID="83154825-18ec-4193-8aff-3983a8c47219" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s" Oct 30 06:03:38.426379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.372140 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.255:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:38.426379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.385379 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:03:38.475595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.475555 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" event={"ID":"af19d653-5b0a-4e26-9609-31d9065f59cc","Type":"ContainerStarted","Data":"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6"} Oct 30 06:03:38.476558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.476512 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-862dc2-6zzgb"] Oct 30 06:03:38.478461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.478362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:03:38.509559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.497263 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:03:38.566151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.566124 6171 generic.go:356] "Generic (PLEG): container finished" podID="22dec94d-c650-478e-b581-63ae985bca08" containerID="8024a4e23239859786c996d82356b21ba39098ac3fb3891d831fcff44513e185" exitCode=0 Oct 30 06:03:38.566567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.566540 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" event={"ID":"22dec94d-c650-478e-b581-63ae985bca08","Type":"ContainerDied","Data":"8024a4e23239859786c996d82356b21ba39098ac3fb3891d831fcff44513e185"} Oct 30 06:03:38.650322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.650258 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podStartSLOduration=3.650238238 podStartE2EDuration="3.650238238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:38.611372281 +0000 UTC m=+1877.051749474" watchObservedRunningTime="2026-10-30 06:03:38.650238238 +0000 UTC m=+1877.090615433" Oct 30 06:03:38.758297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.758264 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 06:03:38.855259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.854585 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:38.855259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.854651 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:38.861486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.861191 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\") pod \"22dec94d-c650-478e-b581-63ae985bca08\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " Oct 30 06:03:38.861486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.861429 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\" (UniqueName: \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\") pod \"22dec94d-c650-478e-b581-63ae985bca08\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " Oct 30 06:03:38.861691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.861561 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\") pod \"22dec94d-c650-478e-b581-63ae985bca08\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " Oct 30 06:03:38.861691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.861636 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\") pod \"22dec94d-c650-478e-b581-63ae985bca08\" (UID: \"22dec94d-c650-478e-b581-63ae985bca08\") " Oct 30 06:03:38.864085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.861948 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca" pod "22dec94d-c650-478e-b581-63ae985bca08" (UID: "22dec94d-c650-478e-b581-63ae985bca08"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:38.864085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.862358 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/22dec94d-c650-478e-b581-63ae985bca08-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:38.867349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.867320 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn" pod "22dec94d-c650-478e-b581-63ae985bca08" (UID: "22dec94d-c650-478e-b581-63ae985bca08"). InnerVolumeSpecName "kube-api-access-88vfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:38.867584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.867557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs" pod "22dec94d-c650-478e-b581-63ae985bca08" (UID: "22dec94d-c650-478e-b581-63ae985bca08"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:38.869910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.869882 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls" pod "22dec94d-c650-478e-b581-63ae985bca08" (UID: "22dec94d-c650-478e-b581-63ae985bca08"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:38.875289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.875265 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:03:38.876891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.876870 6171 cpu_manager.go:414] "containerMap: removing container" podUID="22dec94d-c650-478e-b581-63ae985bca08" containerName="maas-api" Oct 30 06:03:38.876983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.876894 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22dec94d-c650-478e-b581-63ae985bca08" containerName="maas-api" Oct 30 06:03:38.877176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.877160 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="22dec94d-c650-478e-b581-63ae985bca08" containerName="maas-api" Oct 30 06:03:38.917067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:38.908369 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.054903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.054858 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:03:39.092068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.065423 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-62316c-serving-cert\"" Oct 30 06:03:39.110867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.110788 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88vfn\" (UniqueName: \"kubernetes.io/projected/22dec94d-c650-478e-b581-63ae985bca08-kube-api-access-88vfn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:39.111122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.111101 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:39.111261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.111246 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/22dec94d-c650-478e-b581-63ae985bca08-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:39.130073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.112353 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:39.186067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.177675 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:39.186067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.177746 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:39.247072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.234952 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.247072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.235020 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.247072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.235202 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.250225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.250193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tdqm\" (UniqueName: \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.351916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.351875 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.352584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.352561 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tdqm\" (UniqueName: \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.352887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.352871 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.353061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.353026 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.353542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.353514 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.358066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.355331 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.358066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.355678 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.377834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.377772 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tdqm\" (UniqueName: \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\") pod \"maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.478783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.478750 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:39.619082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.614919 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" Oct 30 06:03:39.620428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.620399 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh" event={"ID":"22dec94d-c650-478e-b581-63ae985bca08","Type":"ContainerDied","Data":"26d6723865b4c930d6265e28e8411af6c368642a837f9d2a7f75bed85fcce559"} Oct 30 06:03:39.620682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.620650 6171 scope.go:122] "RemoveContainer" containerID="8024a4e23239859786c996d82356b21ba39098ac3fb3891d831fcff44513e185" Oct 30 06:03:39.767285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.767252 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc31b1d4-8715-432d-acc3-25aa3b912cf8" path="/var/lib/kubelet/pods/fc31b1d4-8715-432d-acc3-25aa3b912cf8/volumes" Oct 30 06:03:39.768279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.768256 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:03:39.793964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.793919 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:03:39.794370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.794344 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.831404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.831376 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aee070\"/\"kube-root-ca.crt\"" Oct 30 06:03:39.842079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.832290 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aee070\"/\"test-model-aee070-kserve-self-signed-certs\"" Oct 30 06:03:39.855511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.846754 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-aee070\"/\"openshift-service-ca.crt\"" Oct 30 06:03:39.855710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.846856 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-aee070\"/\"default-dockercfg-pdlqw\"" Oct 30 06:03:39.855800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.847398 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 06:03:39.855897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.847961 6171 patch_prober.go:29] interesting pod/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" start-of-body= Oct 30 06:03:39.856058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.856023 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.241:15021/healthz/ready\": dial tcp 10.133.0.241:15021: connect: connection refused" Oct 30 06:03:39.861347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.861303 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-862dc2-69c5dffb64-p4knh"] Oct 30 06:03:39.868379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.868356 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.868683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.868664 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.869493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.869471 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.869812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.869794 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.870111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.870090 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5h9\" (UniqueName: \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:39.870343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.870324 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.978497 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.978610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.979553 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.997577 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.997703 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.997826 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5h9\" (UniqueName: \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.021189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:39.997881 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.069068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.065428 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.080327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.071189 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.137069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.130655 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:03:40.181442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.180866 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.187064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.185460 6171 patch_prober.go:29] interesting pod/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" start-of-body= Oct 30 06:03:40.187064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.185513 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.242:15021/healthz/ready\": dial tcp 10.133.0.242:15021: connect: connection refused" Oct 30 06:03:40.240774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.195977 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.240774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.208868 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5h9\" (UniqueName: \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\") pod \"test-model-aee070-kserve-6d67558475-psqdc\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.447377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.447286 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:40.635866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.635772 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" event={"ID":"6ef33713-6170-4f22-9560-7f04b24c419f","Type":"ContainerStarted","Data":"2cdf199e74113babc2fa3c4cf21c2e629364d48c011dcd90b368c4cbceb46fb7"} Oct 30 06:03:40.785793 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:40.785759 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9b8c1f7b_4e98_4fa9_96f6_69e6e4b038af.slice/crio-856d881de65d79e77526f23899c84319329c6dce1966421cb2e44fa2cce36f89 WatchSource:0}: Error finding container 856d881de65d79e77526f23899c84319329c6dce1966421cb2e44fa2cce36f89: Status 404 returned error can't find the container with id 856d881de65d79e77526f23899c84319329c6dce1966421cb2e44fa2cce36f89 Oct 30 06:03:40.788914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.788890 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:03:40.854593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.854570 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:03:40.854891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.854870 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:03:40.855636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:40.855619 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:03:41.177368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.177336 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:03:41.177842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.177827 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:03:41.178532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.178510 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:03:41.646954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.646907 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" event={"ID":"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af","Type":"ContainerStarted","Data":"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8"} Oct 30 06:03:41.646954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.646956 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" event={"ID":"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af","Type":"ContainerStarted","Data":"856d881de65d79e77526f23899c84319329c6dce1966421cb2e44fa2cce36f89"} Oct 30 06:03:41.649299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.649261 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" event={"ID":"6ef33713-6170-4f22-9560-7f04b24c419f","Type":"ContainerStarted","Data":"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2"} Oct 30 06:03:41.649479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.649362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:41.724078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.723989 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podStartSLOduration=2.723968991 podStartE2EDuration="2.723968991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:41.717126962 +0000 UTC m=+1880.157504217" watchObservedRunningTime="2026-10-30 06:03:41.723968991 +0000 UTC m=+1880.164346177" Oct 30 06:03:41.801809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:41.801775 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22dec94d-c650-478e-b581-63ae985bca08" path="/var/lib/kubelet/pods/22dec94d-c650-478e-b581-63ae985bca08/volumes" Oct 30 06:03:42.447827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:42.447797 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:45.597977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:45.597906 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" podStartSLOduration=72.10997683 podStartE2EDuration="1m12.597885129s" totalImagesPullingTime="487.908299ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:41.774163237 +0000 UTC m=+1880.214540455" watchObservedRunningTime="2026-10-30 06:03:45.597885129 +0000 UTC m=+1884.038262323" Oct 30 06:03:45.616110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:45.616085 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 06:03:45.616326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:45.616300 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" containerID="cri-o://3827af6668b6aa61d86628baa9eea367d3c2fbc628922f0b7f9d6220515c086f" gracePeriod=30 Oct 30 06:03:46.034402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.034363 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:46.066841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.066783 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:46.181592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.181560 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk"] Oct 30 06:03:46.186724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.186704 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.203635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.203613 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:03:46.203903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.203871 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:03:46.206144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.206121 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk"] Oct 30 06:03:46.297209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.297114 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.297375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.297350 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pj2n\" (UniqueName: \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.297554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.297534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.398971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.398936 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.399206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.399187 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.399423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.399404 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pj2n\" (UniqueName: \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.399548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.399527 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.401716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.401691 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.432723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.432698 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pj2n\" (UniqueName: \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\") pod \"maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.503697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.503673 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:46.833201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.833173 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:03:46.837214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:46.837190 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk"] Oct 30 06:03:47.676669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:47.676642 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:03:47.836843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:47.836812 6171 generic.go:356] "Generic (PLEG): container finished" podID="e61787ad-8cec-40e9-80c1-27553354b2f3" containerID="e1fe214be631f1d8fdd02a072c6a184fc2c0f012ee371b34b7d4c38dc1813956" exitCode=0 Oct 30 06:03:47.837307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:47.836941 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" event={"ID":"e61787ad-8cec-40e9-80c1-27553354b2f3","Type":"ContainerDied","Data":"e1fe214be631f1d8fdd02a072c6a184fc2c0f012ee371b34b7d4c38dc1813956"} Oct 30 06:03:47.837307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:47.836995 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" event={"ID":"e61787ad-8cec-40e9-80c1-27553354b2f3","Type":"ContainerStarted","Data":"edf5ce20ece39ff32209c53000790d6e02cde01aa34cbcb7dd0e4b8e12d173a0"} Oct 30 06:03:49.027349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.027326 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:49.156489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.156406 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\") pod \"e61787ad-8cec-40e9-80c1-27553354b2f3\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " Oct 30 06:03:49.156666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.156585 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\") pod \"e61787ad-8cec-40e9-80c1-27553354b2f3\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " Oct 30 06:03:49.156666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.156628 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\" (UniqueName: \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\") pod \"e61787ad-8cec-40e9-80c1-27553354b2f3\" (UID: \"e61787ad-8cec-40e9-80c1-27553354b2f3\") " Oct 30 06:03:49.156797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.156768 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca" pod "e61787ad-8cec-40e9-80c1-27553354b2f3" (UID: "e61787ad-8cec-40e9-80c1-27553354b2f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:49.157445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.157423 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:49.158796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.158769 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca" pod "e61787ad-8cec-40e9-80c1-27553354b2f3" (UID: "e61787ad-8cec-40e9-80c1-27553354b2f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:03:49.158912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.158897 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n" pod "e61787ad-8cec-40e9-80c1-27553354b2f3" (UID: "e61787ad-8cec-40e9-80c1-27553354b2f3"). InnerVolumeSpecName "kube-api-access-2pj2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:49.259040 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.259015 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e61787ad-8cec-40e9-80c1-27553354b2f3-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:49.259040 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.259056 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pj2n\" (UniqueName: \"kubernetes.io/projected/e61787ad-8cec-40e9-80c1-27553354b2f3-kube-api-access-2pj2n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:49.857104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.857031 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" Oct 30 06:03:49.857104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.857094 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk" event={"ID":"e61787ad-8cec-40e9-80c1-27553354b2f3","Type":"ContainerDied","Data":"edf5ce20ece39ff32209c53000790d6e02cde01aa34cbcb7dd0e4b8e12d173a0"} Oct 30 06:03:49.857305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:49.857145 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edf5ce20ece39ff32209c53000790d6e02cde01aa34cbcb7dd0e4b8e12d173a0" Oct 30 06:03:50.447706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.447676 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:50.461091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.461030 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:50.876544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.876504 6171 generic.go:356] "Generic (PLEG): container finished" podID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerID="3827af6668b6aa61d86628baa9eea367d3c2fbc628922f0b7f9d6220515c086f" exitCode=0 Oct 30 06:03:50.876706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.876570 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerDied","Data":"3827af6668b6aa61d86628baa9eea367d3c2fbc628922f0b7f9d6220515c086f"} Oct 30 06:03:50.876706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.876634 6171 scope.go:122] "RemoveContainer" containerID="9305a9e20ea8bc0efed20121f782dcadf74b0485f4b769569c594315fb3717e4" Oct 30 06:03:50.891238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.891215 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:03:50.923745 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:50.923726 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 06:03:51.099799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.099715 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.099983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.099858 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.099983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.099922 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.099983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.099957 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100095 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100191 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:51.100221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100206 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:51.100221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100218 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100387 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100413 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:51.100722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100537 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100613 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\") pod \"9358b27a-2ed7-4acf-b094-b21d2be598ac\" (UID: \"9358b27a-2ed7-4acf-b094-b21d2be598ac\") " Oct 30 06:03:51.100722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.100601 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:51.101090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101000 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:03:51.101919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101897 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.102002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101929 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.102002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101946 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.102002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101960 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9358b27a-2ed7-4acf-b094-b21d2be598ac-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.102002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.101974 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.102528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.102504 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:51.102822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.102806 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "kube-api-access-lqrtc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:03:51.102932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.102919 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:03:51.103114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.103099 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo" pod "9358b27a-2ed7-4acf-b094-b21d2be598ac" (UID: "9358b27a-2ed7-4acf-b094-b21d2be598ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:03:51.204254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.204226 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.204254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.204253 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.204254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.204261 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9358b27a-2ed7-4acf-b094-b21d2be598ac-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.204506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.204270 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqrtc\" (UniqueName: \"kubernetes.io/projected/9358b27a-2ed7-4acf-b094-b21d2be598ac-kube-api-access-lqrtc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:03:51.913511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.913477 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" Oct 30 06:03:51.913511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.913482 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr" event={"ID":"9358b27a-2ed7-4acf-b094-b21d2be598ac","Type":"ContainerDied","Data":"d593d4ff0803a2dba8a1071839fc454f42aadb1409d1ad27ee380ed40bde9b85"} Oct 30 06:03:51.914074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:51.913596 6171 scope.go:122] "RemoveContainer" containerID="3827af6668b6aa61d86628baa9eea367d3c2fbc628922f0b7f9d6220515c086f" Oct 30 06:03:52.021008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:52.020982 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 06:03:52.026673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:52.026646 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-862dc2-openshift-default-66cbdd88d9-kwwnr"] Oct 30 06:03:53.675112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:53.675079 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" path="/var/lib/kubelet/pods/9358b27a-2ed7-4acf-b094-b21d2be598ac/volumes" Oct 30 06:03:55.247877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.247838 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:03:55.248959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.248936 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:03:55.248959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.248960 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:03:55.249168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249015 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e61787ad-8cec-40e9-80c1-27553354b2f3" containerName="revoke-keys" Oct 30 06:03:55.249168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249026 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e61787ad-8cec-40e9-80c1-27553354b2f3" containerName="revoke-keys" Oct 30 06:03:55.249168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249056 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:03:55.249168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249062 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:03:55.249338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249220 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e61787ad-8cec-40e9-80c1-27553354b2f3" containerName="revoke-keys" Oct 30 06:03:55.249338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.249234 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:03:55.255301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.255278 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.280180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.280154 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c54211\"/\"kube-root-ca.crt\"" Oct 30 06:03:55.280541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.280523 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-c54211\"/\"openshift-service-ca.crt\"" Oct 30 06:03:55.285140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.285095 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:03:55.286690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.286296 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c54211\"/\"default-dockercfg-sb9wx\"" Oct 30 06:03:55.286690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.286402 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-c54211\"/\"test-model-c54211-kserve-self-signed-certs\"" Oct 30 06:03:55.317635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.317596 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjzch\" (UniqueName: \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.317804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.317734 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.317943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.317824 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.317943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.317914 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.318112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.317979 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.318194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.318122 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.418990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.418960 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.418996 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419250 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419462 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjzch\" (UniqueName: \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419571 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419580 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.419983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.419963 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.421406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.421386 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.421748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.421729 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.443862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.443839 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjzch\" (UniqueName: \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\") pod \"test-model-c54211-kserve-5b6b668ff-jd95d\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:55.582669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:55.579705 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:03:56.047062 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:03:56.047017 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2d5bdc_31c7_4fa7_97c2_3435a4f9f8fd.slice/crio-c09ff7f69df1deecb929327aaebd0f66e5f2e686d1840975800fd015b1f6ce27 WatchSource:0}: Error finding container c09ff7f69df1deecb929327aaebd0f66e5f2e686d1840975800fd015b1f6ce27: Status 404 returned error can't find the container with id c09ff7f69df1deecb929327aaebd0f66e5f2e686d1840975800fd015b1f6ce27 Oct 30 06:03:56.048806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:56.048774 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:03:56.383892 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:03:56.383797 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:03:56.992067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:56.991445 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" event={"ID":"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd","Type":"ContainerStarted","Data":"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b"} Oct 30 06:03:56.992067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:56.991519 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" event={"ID":"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd","Type":"ContainerStarted","Data":"c09ff7f69df1deecb929327aaebd0f66e5f2e686d1840975800fd015b1f6ce27"} Oct 30 06:03:57.041922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:57.041863 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podStartSLOduration=2.041849044 podStartE2EDuration="2.041849044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:57.037165218 +0000 UTC m=+1895.477542407" watchObservedRunningTime="2026-10-30 06:03:57.041849044 +0000 UTC m=+1895.482226256" Oct 30 06:03:57.383232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:57.383161 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 06:03:57.581562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:03:57.581531 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:04:02.384150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.384056 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" podUID="83154825-18ec-4193-8aff-3983a8c47219" containerName="payload-processing" containerID="cri-o://1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630" gracePeriod=25 Oct 30 06:04:02.710130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.710104 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 06:04:02.821452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.821419 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\") pod \"83154825-18ec-4193-8aff-3983a8c47219\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " Oct 30 06:04:02.821652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.821617 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\" (UniqueName: \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\") pod \"83154825-18ec-4193-8aff-3983a8c47219\" (UID: \"83154825-18ec-4193-8aff-3983a8c47219\") " Oct 30 06:04:02.821873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.821844 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume" pod "83154825-18ec-4193-8aff-3983a8c47219" (UID: "83154825-18ec-4193-8aff-3983a8c47219"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:02.822935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.822895 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83154825-18ec-4193-8aff-3983a8c47219-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:02.824428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.824405 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn" pod "83154825-18ec-4193-8aff-3983a8c47219" (UID: "83154825-18ec-4193-8aff-3983a8c47219"). InnerVolumeSpecName "kube-api-access-v5lsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:02.924377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:02.924304 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5lsn\" (UniqueName: \"kubernetes.io/projected/83154825-18ec-4193-8aff-3983a8c47219-kube-api-access-v5lsn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:03.105875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.105839 6171 generic.go:356] "Generic (PLEG): container finished" podID="83154825-18ec-4193-8aff-3983a8c47219" containerID="1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630" exitCode=0 Oct 30 06:04:03.106093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.105883 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" Oct 30 06:04:03.106093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.105926 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" event={"ID":"83154825-18ec-4193-8aff-3983a8c47219","Type":"ContainerDied","Data":"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630"} Oct 30 06:04:03.106093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.105986 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx" event={"ID":"83154825-18ec-4193-8aff-3983a8c47219","Type":"ContainerDied","Data":"ff30cef189f2c9b9734a87c7321d1911a4317ca96f90e953e9279cc5a7a038e6"} Oct 30 06:04:03.106093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.106008 6171 scope.go:122] "RemoveContainer" containerID="1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630" Oct 30 06:04:03.122762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.122740 6171 scope.go:122] "RemoveContainer" containerID="1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630" Oct 30 06:04:03.123035 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:03.123015 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630\": container with ID starting with 1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630 not found: ID does not exist" containerID="1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630" Oct 30 06:04:03.123216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.123058 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630"} err="failed to get container status \"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630\": rpc error: code = NotFound desc = could not find container \"1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630\": container with ID starting with 1413d3aec388402214e524e504bbcf61ec8baa0dd544141f75e6e48a8e921630 not found: ID does not exist" Oct 30 06:04:03.168162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.168128 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 06:04:03.182171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.182096 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-de7242-777f988ccf-879xx"] Oct 30 06:04:03.701879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:03.701842 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83154825-18ec-4193-8aff-3983a8c47219" path="/var/lib/kubelet/pods/83154825-18ec-4193-8aff-3983a8c47219/volumes" Oct 30 06:04:05.458231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.458203 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:04:05.459170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.459154 6171 cpu_manager.go:414] "containerMap: removing container" podUID="83154825-18ec-4193-8aff-3983a8c47219" containerName="payload-processing" Oct 30 06:04:05.459228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.459171 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83154825-18ec-4193-8aff-3983a8c47219" containerName="payload-processing" Oct 30 06:04:05.459346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.459337 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9358b27a-2ed7-4acf-b094-b21d2be598ac" containerName="istio-proxy" Oct 30 06:04:05.459382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.459354 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="83154825-18ec-4193-8aff-3983a8c47219" containerName="payload-processing" Oct 30 06:04:05.464745 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.464729 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.485184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.485161 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:04:05.493110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.492940 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-0d64c9-serving-cert\"" Oct 30 06:04:05.514266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.514239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.514366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.514279 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.514430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.514413 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.514547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.514530 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwgsf\" (UniqueName: \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.581413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.581388 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:04:05.593558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.593535 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:04:05.616604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.616571 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.616604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.616600 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.616784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.616663 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.616784 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:05.616715 6171 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-serving-cert: secret "maas-api-e2e-inf-b-0d64c9-serving-cert" not found Oct 30 06:04:05.616784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.616746 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwgsf\" (UniqueName: \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.616784 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:05.616775 6171 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls podName:f5215668-1f63-40b2-9ddb-85095076ef29 nodeName:}" failed. No retries permitted until 2026-10-30 06:04:06.116755272 +0000 UTC m=+1904.557132457 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls") pod "maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" (UID: "f5215668-1f63-40b2-9ddb-85095076ef29") : secret "maas-api-e2e-inf-b-0d64c9-serving-cert" not found Oct 30 06:04:05.617440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.617419 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.619384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.619365 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.643981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.643239 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwgsf\" (UniqueName: \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:05.814509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.814475 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:04:05.820118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.820096 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.852160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.852134 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7ad9ad\"/\"kube-root-ca.crt\"" Oct 30 06:04:05.853115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.853083 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-7ad9ad\"/\"openshift-service-ca.crt\"" Oct 30 06:04:05.853234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.853218 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7ad9ad\"/\"test-model-7ad9ad-kserve-self-signed-certs\"" Oct 30 06:04:05.853426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.853406 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-7ad9ad\"/\"default-dockercfg-thlzr\"" Oct 30 06:04:05.862090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.858304 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:04:05.920942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.920912 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkdg\" (UniqueName: \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.921147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.921100 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.921200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.921143 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.921293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.921271 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.921340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.921321 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:05.921379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:05.921340 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023304 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023407 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023444 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023483 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023739 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023760 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkdg\" (UniqueName: \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023817 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023764 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.023962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.023945 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.026261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.026239 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.026361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.026316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.053426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.053401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkdg\" (UniqueName: \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\") pod \"test-model-7ad9ad-kserve-749f974487-lqgcp\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.172127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.171581 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:06.182677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.182642 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:06.316805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.316762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") pod \"maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:06.398200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.387687 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:06.406300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:06.405093 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:04:07.016470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:07.014097 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:04:07.016470 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:07.014811 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod202a3a3b_fb27_4a44_85b7_ccac61e07326.slice/crio-4f75e19fe3178f03f0227093eab06cc4d95dc887089b548c3a66889ff2317e68 WatchSource:0}: Error finding container 4f75e19fe3178f03f0227093eab06cc4d95dc887089b548c3a66889ff2317e68: Status 404 returned error can't find the container with id 4f75e19fe3178f03f0227093eab06cc4d95dc887089b548c3a66889ff2317e68 Oct 30 06:04:07.114395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:07.113647 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:04:07.333137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:07.332196 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" event={"ID":"202a3a3b-fb27-4a44-85b7-ccac61e07326","Type":"ContainerStarted","Data":"4f75e19fe3178f03f0227093eab06cc4d95dc887089b548c3a66889ff2317e68"} Oct 30 06:04:07.333911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:07.333885 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" event={"ID":"f5215668-1f63-40b2-9ddb-85095076ef29","Type":"ContainerStarted","Data":"b64549941d64ec94d4e4d642973e0146579c591f7c520304fd7a1058b942c34d"} Oct 30 06:04:08.349819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:08.349725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" event={"ID":"202a3a3b-fb27-4a44-85b7-ccac61e07326","Type":"ContainerStarted","Data":"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1"} Oct 30 06:04:08.351661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:08.351621 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" event={"ID":"f5215668-1f63-40b2-9ddb-85095076ef29","Type":"ContainerStarted","Data":"3f28ad3192a9c920b318619bbbd8cca06e8187fcd0f2e42f005d14b8d6c3c997"} Oct 30 06:04:08.351776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:08.351708 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:08.427009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:08.426882 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podStartSLOduration=3.426864252 podStartE2EDuration="3.426864252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:08.421706461 +0000 UTC m=+1906.862083653" watchObservedRunningTime="2026-10-30 06:04:08.426864252 +0000 UTC m=+1906.867241446" Oct 30 06:04:08.485877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:08.485813 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" podStartSLOduration=2.842916804 podStartE2EDuration="3.485794545s" totalImagesPullingTime="642.877741ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:08.480572495 +0000 UTC m=+1906.920949688" watchObservedRunningTime="2026-10-30 06:04:08.485794545 +0000 UTC m=+1906.926171738" Oct 30 06:04:09.184516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:09.184478 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:14.368459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:14.368427 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:04:16.184760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:16.184725 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:16.197205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:16.197182 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:16.483286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:16.483259 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:04:21.226710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.226676 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:04:21.234103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.234073 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.256372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.256343 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:04:21.272379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.271888 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1d09f\"/\"openshift-service-ca.crt\"" Oct 30 06:04:21.278792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.274140 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1d09f\"/\"test-model-c1d09f-kserve-self-signed-certs\"" Oct 30 06:04:21.278792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.274803 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-c1d09f\"/\"default-dockercfg-9qsgl\"" Oct 30 06:04:21.278792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.274874 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-c1d09f\"/\"kube-root-ca.crt\"" Oct 30 06:04:21.332290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332256 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.332456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332317 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.332505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332468 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.332505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.332585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332553 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.332682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.332663 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434178 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434268 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434388 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434419 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434480 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434553 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434596 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434661 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.434771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.434753 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.436948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.436924 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.437295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.437275 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.548117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.531902 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\") pod \"test-model-c1d09f-kserve-5598b7b945-chljq\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:21.574427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:21.574118 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:22.051139 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:22.051108 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfd940f31_3819_4f9b_9769_6d9f3e02e31c.slice/crio-7be2d545264c674f837c1015599428691644a6d73548cba49aa0c32464f0a7a6 WatchSource:0}: Error finding container 7be2d545264c674f837c1015599428691644a6d73548cba49aa0c32464f0a7a6: Status 404 returned error can't find the container with id 7be2d545264c674f837c1015599428691644a6d73548cba49aa0c32464f0a7a6 Oct 30 06:04:22.052810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:22.052786 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:04:22.585458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:22.585418 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" event={"ID":"fd940f31-3819-4f9b-9769-6d9f3e02e31c","Type":"ContainerStarted","Data":"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261"} Oct 30 06:04:22.585888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:22.585477 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" event={"ID":"fd940f31-3819-4f9b-9769-6d9f3e02e31c","Type":"ContainerStarted","Data":"7be2d545264c674f837c1015599428691644a6d73548cba49aa0c32464f0a7a6"} Oct 30 06:04:22.634113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:22.634064 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podStartSLOduration=1.634032294 podStartE2EDuration="1.634032294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:22.630299641 +0000 UTC m=+1921.070676836" watchObservedRunningTime="2026-10-30 06:04:22.634032294 +0000 UTC m=+1921.074409485" Oct 30 06:04:23.575498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:23.575462 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:26.032309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:26.032269 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp"] Oct 30 06:04:26.046968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:26.046936 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db02f378-2kmsp"] Oct 30 06:04:26.382618 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:26.382535 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:04:26.405900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:26.405873 6171 scope.go:122] "RemoveContainer" containerID="f4b74026151511ae6c9559ce0adc30418ed16098f275457d969ff06f41d4af13" Oct 30 06:04:27.675725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:27.675691 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc26d67-28d8-41ee-a77f-11d2ab469596" path="/var/lib/kubelet/pods/cfc26d67-28d8-41ee-a77f-11d2ab469596/volumes" Oct 30 06:04:30.590186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:30.590151 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 06:04:30.590984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:30.590953 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" podUID="6ce723f5-1758-4ccf-bd0b-426238b0d167" containerName="maas-api" containerID="cri-o://005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f" gracePeriod=30 Oct 30 06:04:31.458820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.458791 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 06:04:31.551629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.551597 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\") pod \"6ce723f5-1758-4ccf-bd0b-426238b0d167\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " Oct 30 06:04:31.551789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.551703 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\" (UniqueName: \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\") pod \"6ce723f5-1758-4ccf-bd0b-426238b0d167\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " Oct 30 06:04:31.551789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.551775 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\") pod \"6ce723f5-1758-4ccf-bd0b-426238b0d167\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " Oct 30 06:04:31.551920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.551903 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\") pod \"6ce723f5-1758-4ccf-bd0b-426238b0d167\" (UID: \"6ce723f5-1758-4ccf-bd0b-426238b0d167\") " Oct 30 06:04:31.552295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.552268 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca" pod "6ce723f5-1758-4ccf-bd0b-426238b0d167" (UID: "6ce723f5-1758-4ccf-bd0b-426238b0d167"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:31.552877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.552855 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ce723f5-1758-4ccf-bd0b-426238b0d167-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:31.554005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.553984 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l" pod "6ce723f5-1758-4ccf-bd0b-426238b0d167" (UID: "6ce723f5-1758-4ccf-bd0b-426238b0d167"). InnerVolumeSpecName "kube-api-access-pf97l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:31.554456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.554436 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls" pod "6ce723f5-1758-4ccf-bd0b-426238b0d167" (UID: "6ce723f5-1758-4ccf-bd0b-426238b0d167"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:31.554456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.554446 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs" pod "6ce723f5-1758-4ccf-bd0b-426238b0d167" (UID: "6ce723f5-1758-4ccf-bd0b-426238b0d167"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:31.575211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.575189 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:31.587771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.587748 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:31.588399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.588385 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk"] Oct 30 06:04:31.594181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.594128 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-de7242-l7dhk"] Oct 30 06:04:31.658237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.658208 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:31.658380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.658248 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf97l\" (UniqueName: \"kubernetes.io/projected/6ce723f5-1758-4ccf-bd0b-426238b0d167-kube-api-access-pf97l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:31.658380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.658264 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ce723f5-1758-4ccf-bd0b-426238b0d167-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:31.736865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.735278 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e61787ad-8cec-40e9-80c1-27553354b2f3" path="/var/lib/kubelet/pods/e61787ad-8cec-40e9-80c1-27553354b2f3/volumes" Oct 30 06:04:31.758286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.758234 6171 generic.go:356] "Generic (PLEG): container finished" podID="6ce723f5-1758-4ccf-bd0b-426238b0d167" containerID="005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f" exitCode=0 Oct 30 06:04:31.760154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.760117 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" Oct 30 06:04:31.782033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.771994 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" event={"ID":"6ce723f5-1758-4ccf-bd0b-426238b0d167","Type":"ContainerDied","Data":"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f"} Oct 30 06:04:31.795199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.795164 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp" event={"ID":"6ce723f5-1758-4ccf-bd0b-426238b0d167","Type":"ContainerDied","Data":"990ec9b6e161dcb676791c652e03c7ed108b0ed841eb404f58ebf370a20e43f8"} Oct 30 06:04:31.795835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.795434 6171 scope.go:122] "RemoveContainer" containerID="005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f" Oct 30 06:04:31.834569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.834545 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:04:31.839413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.839394 6171 scope.go:122] "RemoveContainer" containerID="005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f" Oct 30 06:04:31.839720 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:31.839703 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f\": container with ID starting with 005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f not found: ID does not exist" containerID="005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f" Oct 30 06:04:31.839802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.839726 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f"} err="failed to get container status \"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f\": rpc error: code = NotFound desc = could not find container \"005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f\": container with ID starting with 005b1ec855442e726b826a15a1e410d65c2b2199cc0c38cb9c4cfa5392d4ad6f not found: ID does not exist" Oct 30 06:04:31.874509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.874450 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 06:04:31.880354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:31.880318 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-de7242-6c494cd77d-m9qzp"] Oct 30 06:04:32.304699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.304661 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:04:32.305927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.305904 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6ce723f5-1758-4ccf-bd0b-426238b0d167" containerName="maas-api" Oct 30 06:04:32.305927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.305926 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ce723f5-1758-4ccf-bd0b-426238b0d167" containerName="maas-api" Oct 30 06:04:32.306174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.306161 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ce723f5-1758-4ccf-bd0b-426238b0d167" containerName="maas-api" Oct 30 06:04:32.311776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.311757 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.333689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.333663 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:04:32.377599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.377574 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-3cc570-openshift-default-dockercfg-svxdq\"" Oct 30 06:04:32.399298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399271 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399374 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399399 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399516 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399551 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399658 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399692 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g4df\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.399773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399721 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.400004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.399982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.501855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.501815 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.501855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.501864 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.501924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.501951 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502099 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502136 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502210 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502233 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g4df\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502263 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502327 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502503 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502517 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502677 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.502798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.502780 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.504675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.504645 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.504884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.504868 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.530131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.530112 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.530339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.530319 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g4df\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\") pod \"e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.630568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.630504 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:32.961516 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:32.961487 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbb7c37a_6b95_4370_979e_3d8f261ae7f3.slice/crio-c3c964c638ff515842c3e90f2f9ba659939049fe1617820f264d54f0619611ef WatchSource:0}: Error finding container c3c964c638ff515842c3e90f2f9ba659939049fe1617820f264d54f0619611ef: Status 404 returned error can't find the container with id c3c964c638ff515842c3e90f2f9ba659939049fe1617820f264d54f0619611ef Oct 30 06:04:32.962788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.962766 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:04:32.964688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.964650 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:32.964793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.964728 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:32.964793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:32.964772 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:33.692070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:33.681015 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ce723f5-1758-4ccf-bd0b-426238b0d167" path="/var/lib/kubelet/pods/6ce723f5-1758-4ccf-bd0b-426238b0d167/volumes" Oct 30 06:04:33.811190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:33.811147 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" event={"ID":"fbb7c37a-6b95-4370-979e-3d8f261ae7f3","Type":"ContainerStarted","Data":"9e0e8c0aae924be81827c7d601faddc15ea1def5d8848eac10afe46789a4dd52"} Oct 30 06:04:33.811400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:33.811210 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" event={"ID":"fbb7c37a-6b95-4370-979e-3d8f261ae7f3","Type":"ContainerStarted","Data":"c3c964c638ff515842c3e90f2f9ba659939049fe1617820f264d54f0619611ef"} Oct 30 06:04:33.879418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:33.879368 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podStartSLOduration=1.8793522230000002 podStartE2EDuration="1.879352223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:33.875560139 +0000 UTC m=+1932.315937337" watchObservedRunningTime="2026-10-30 06:04:33.879352223 +0000 UTC m=+1932.319729415" Oct 30 06:04:34.631652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:34.631617 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:34.633529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:34.633503 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:34.633690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:34.633556 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:35.630799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:35.630761 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:35.631210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:35.630812 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:36.633070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:36.630940 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:36.633070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:36.631002 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:37.631509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:37.631467 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:37.631720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:37.631521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:38.631517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:38.631482 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:38.632017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:38.631554 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:39.631074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:39.631025 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:04:39.631292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:39.631097 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.10:15021/healthz/ready\": dial tcp 10.133.1.10:15021: connect: connection refused" Oct 30 06:04:39.706961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:39.706101 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 06:04:39.706961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:39.706371 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" containerID="cri-o://28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf" gracePeriod=30 Oct 30 06:04:40.637036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:40.637008 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:40.637391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:40.637369 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:40.638027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:40.638009 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:04:41.455325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:41.455292 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:04:41.456585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:41.456126 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" podUID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" containerName="maas-api" containerID="cri-o://fc386c63cbb20e4ddbf68faf0f53d3696fb712d97900a1af5b6c7d488b7db94e" gracePeriod=30 Oct 30 06:04:42.447191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.437758 6171 patch_prober.go:29] interesting pod/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:04:42.447191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.437829 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:04:42.455405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.455086 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x"] Oct 30 06:04:42.461639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.460479 6171 generic.go:356] "Generic (PLEG): container finished" podID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" containerID="fc386c63cbb20e4ddbf68faf0f53d3696fb712d97900a1af5b6c7d488b7db94e" exitCode=0 Oct 30 06:04:42.461639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.460578 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" event={"ID":"03ca92b1-aeed-41e3-bfc3-2f63026b90c1","Type":"ContainerDied","Data":"fc386c63cbb20e4ddbf68faf0f53d3696fb712d97900a1af5b6c7d488b7db94e"} Oct 30 06:04:42.464937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.464914 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-f78ca5-2hq4x"] Oct 30 06:04:42.499093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.499070 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:04:42.539583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.539559 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\") pod \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " Oct 30 06:04:42.539862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.539837 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\") pod \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " Oct 30 06:04:42.539981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.539912 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\") pod \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " Oct 30 06:04:42.539981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.539974 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\" (UniqueName: \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\") pod \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\" (UID: \"03ca92b1-aeed-41e3-bfc3-2f63026b90c1\") " Oct 30 06:04:42.540179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.540157 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca" pod "03ca92b1-aeed-41e3-bfc3-2f63026b90c1" (UID: "03ca92b1-aeed-41e3-bfc3-2f63026b90c1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:42.540898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.540874 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:42.542191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.542169 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls" pod "03ca92b1-aeed-41e3-bfc3-2f63026b90c1" (UID: "03ca92b1-aeed-41e3-bfc3-2f63026b90c1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:42.542270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.542256 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs" pod "03ca92b1-aeed-41e3-bfc3-2f63026b90c1" (UID: "03ca92b1-aeed-41e3-bfc3-2f63026b90c1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:04:42.542356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.542333 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw" pod "03ca92b1-aeed-41e3-bfc3-2f63026b90c1" (UID: "03ca92b1-aeed-41e3-bfc3-2f63026b90c1"). InnerVolumeSpecName "kube-api-access-9zjhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:42.642781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.642741 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:42.642781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.642774 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zjhw\" (UniqueName: \"kubernetes.io/projected/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-kube-api-access-9zjhw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:42.642781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:42.642786 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03ca92b1-aeed-41e3-bfc3-2f63026b90c1-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:43.471722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.471693 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" Oct 30 06:04:43.471722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.471710 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9" event={"ID":"03ca92b1-aeed-41e3-bfc3-2f63026b90c1","Type":"ContainerDied","Data":"79ced74400f342668602c410b413b5eaf957c91079a9c2e33f47c3e494eddf62"} Oct 30 06:04:43.472289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.471811 6171 scope.go:122] "RemoveContainer" containerID="fc386c63cbb20e4ddbf68faf0f53d3696fb712d97900a1af5b6c7d488b7db94e" Oct 30 06:04:43.560025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.559997 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:04:43.574095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.574060 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-f78ca5-644d695967-6bll9"] Oct 30 06:04:43.675945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.675914 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" path="/var/lib/kubelet/pods/03ca92b1-aeed-41e3-bfc3-2f63026b90c1/volumes" Oct 30 06:04:43.676564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:43.676540 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f788b086-a6f9-4955-aa30-36cf491a9eda" path="/var/lib/kubelet/pods/f788b086-a6f9-4955-aa30-36cf491a9eda/volumes" Oct 30 06:04:45.020420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.020392 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 06:04:45.103363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103326 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103399 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103500 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103609 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103637 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103663 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.103693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103699 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.104314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103754 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.104314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103855 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.104314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.103922 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\") pod \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\" (UID: \"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d\") " Oct 30 06:04:45.104547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.104376 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:45.104547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.104497 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.104714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.104692 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.105322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.105297 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.105439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.105326 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.105439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.105342 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.105439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.105356 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.105439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.105370 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.106730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.106692 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:45.108452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.108425 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:45.111488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.111462 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "kube-api-access-ljjw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:45.114883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.114861 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo" pod "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" (UID: "41cf7f40-7748-4be6-bedc-be1c7e3b7b1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:04:45.206883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.206857 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.206883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.206881 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.206883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.206890 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.207142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.206900 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljjw6\" (UniqueName: \"kubernetes.io/projected/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d-kube-api-access-ljjw6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:45.494663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.494629 6171 generic.go:356] "Generic (PLEG): container finished" podID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerID="28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf" exitCode=0 Oct 30 06:04:45.494887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.494679 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" Oct 30 06:04:45.494887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.494709 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerDied","Data":"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf"} Oct 30 06:04:45.494887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.494774 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z" event={"ID":"41cf7f40-7748-4be6-bedc-be1c7e3b7b1d","Type":"ContainerDied","Data":"16a030fef44d38f32a0b855a76d7d1767480cb8dd5d8be5fd5209640db12cb2e"} Oct 30 06:04:45.494887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.494804 6171 scope.go:122] "RemoveContainer" containerID="28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf" Oct 30 06:04:45.508632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.508603 6171 scope.go:122] "RemoveContainer" containerID="9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83" Oct 30 06:04:45.520714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.520694 6171 scope.go:122] "RemoveContainer" containerID="28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf" Oct 30 06:04:45.520983 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:45.520961 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf\": container with ID starting with 28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf not found: ID does not exist" containerID="28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf" Oct 30 06:04:45.521087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.520992 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf"} err="failed to get container status \"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf\": rpc error: code = NotFound desc = could not find container \"28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf\": container with ID starting with 28e650919ac018e25888c8e00aa7395a946855ccfab3958121bfb004101773cf not found: ID does not exist" Oct 30 06:04:45.521087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.521008 6171 scope.go:122] "RemoveContainer" containerID="9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83" Oct 30 06:04:45.521249 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:45.521230 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83\": container with ID starting with 9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83 not found: ID does not exist" containerID="9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83" Oct 30 06:04:45.521288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.521255 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83"} err="failed to get container status \"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83\": rpc error: code = NotFound desc = could not find container \"9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83\": container with ID starting with 9676b739b45170e818b6392f37042dbeac7a25fa5d673c9912fcdfc223c67f83 not found: ID does not exist" Oct 30 06:04:45.557025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.556998 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 06:04:45.571339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.571314 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-de7242-openshift-default-7cf5999677-tdn2z"] Oct 30 06:04:45.677376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:45.677339 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" path="/var/lib/kubelet/pods/41cf7f40-7748-4be6-bedc-be1c7e3b7b1d/volumes" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.547038 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618790 6171 cpu_manager.go:414] "containerMap: removing container" podUID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" containerName="maas-api" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618815 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" containerName="maas-api" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618840 6171 cpu_manager.go:414] "containerMap: removing container" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618848 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618929 6171 cpu_manager.go:414] "containerMap: removing container" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.618938 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.619184 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="03ca92b1-aeed-41e3-bfc3-2f63026b90c1" containerName="maas-api" Oct 30 06:04:49.624164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.619206 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:49.628008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.627986 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:04:49.628785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.628331 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.668133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.651680 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-62316c\"/\"openshift-service-ca.crt\"" Oct 30 06:04:49.668133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.667265 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-62316c\"/\"kube-root-ca.crt\"" Oct 30 06:04:49.685310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.671365 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-62316c\"/\"default-dockercfg-rbqlm\"" Oct 30 06:04:49.685310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.671463 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-62316c\"/\"test-model-62316c-kserve-self-signed-certs\"" Oct 30 06:04:49.752145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752111 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.752343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752155 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6xt9\" (UniqueName: \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.752343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752276 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.752343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752305 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.752884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752870 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.752965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.752922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.855068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.854741 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.855068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.855070 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.855363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.855115 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6xt9\" (UniqueName: \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.855363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.855244 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.855363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.855280 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.892787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.855575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.892787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.858437 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.892787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.858961 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.892787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.859866 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.896438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.896406 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6xt9\" (UniqueName: \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.931068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.930583 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.935093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.933764 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\") pod \"test-model-62316c-kserve-6bd459fb77-t4d7p\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:49.999459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:49.999420 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:50.506631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.506597 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:04:50.692907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.692872 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:04:50.693862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.693814 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" containerID="cri-o://6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e" gracePeriod=30 Oct 30 06:04:50.698160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.698120 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" event={"ID":"ffc9eee7-3498-464d-bff5-82b4d48a1360","Type":"ContainerStarted","Data":"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b"} Oct 30 06:04:50.698282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.698174 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" event={"ID":"ffc9eee7-3498-464d-bff5-82b4d48a1360","Type":"ContainerStarted","Data":"f4612e9e903d37fcc07fd0cc3f2f1527df7ecf847142c8b883d63fbab60bf847"} Oct 30 06:04:50.908068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:50.883682 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podStartSLOduration=1.8836588650000001 podStartE2EDuration="1.883658865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:50.808621931 +0000 UTC m=+1949.248999126" watchObservedRunningTime="2026-10-30 06:04:50.883658865 +0000 UTC m=+1949.324036060" Oct 30 06:04:51.009069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:51.000164 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:04:51.993221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:51.993183 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:04:51.994599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:51.994575 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="41cf7f40-7748-4be6-bedc-be1c7e3b7b1d" containerName="istio-proxy" Oct 30 06:04:51.999027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:51.999006 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.017092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.016843 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-d3e7e6-openshift-default-dockercfg-vchrg\"" Oct 30 06:04:52.036724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.036693 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:04:52.200218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.190597 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.200218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.190735 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.200218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.190853 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbvl\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.200218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.190932 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.200218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.190975 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.209082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.207104 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.209082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.207239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.209082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.207282 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.209082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.207328 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.309714 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.309780 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.309820 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.309895 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.309997 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.310124 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbvl\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.310200 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.310241 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.310428 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.311326 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.311616 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.313387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.311842 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.327130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.315019 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.327806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.327775 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.339855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.339760 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.353106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.353016 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.354824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.354791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbvl\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.380065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.380008 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\") pod \"e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.622118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.621281 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:52.983417 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:52.983377 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae869cc6_d981_4fe6_86ca_36a4eabd6b95.slice/crio-7d1a71dd4eb74ea024f51524bf96f4f33ea59893fbe87d02e0a29c29b4396645 WatchSource:0}: Error finding container 7d1a71dd4eb74ea024f51524bf96f4f33ea59893fbe87d02e0a29c29b4396645: Status 404 returned error can't find the container with id 7d1a71dd4eb74ea024f51524bf96f4f33ea59893fbe87d02e0a29c29b4396645 Oct 30 06:04:52.985222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.985199 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:04:52.986839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.986807 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:52.986931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.986881 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:52.986931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:52.986919 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:04:53.737335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:53.737291 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerStarted","Data":"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2"} Oct 30 06:04:53.737764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:53.737359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerStarted","Data":"7d1a71dd4eb74ea024f51524bf96f4f33ea59893fbe87d02e0a29c29b4396645"} Oct 30 06:04:53.800861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:53.800808 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podStartSLOduration=2.8007881919999997 podStartE2EDuration="2.800788192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:53.794636567 +0000 UTC m=+1952.235013785" watchObservedRunningTime="2026-10-30 06:04:53.800788192 +0000 UTC m=+1952.241165386" Oct 30 06:04:54.623076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:54.623015 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:54.630638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:54.630609 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:54.747340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:54.747310 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:54.748310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:54.748286 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:04:56.050978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.050950 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:04:56.104554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104524 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104626 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104692 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104768 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104794 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104906 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.104992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.104930 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.105128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105005 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.105128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105057 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\") pod \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\" (UID: \"5f5963eb-38cb-45a2-9ae8-7e8a60d398da\") " Oct 30 06:04:56.105128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105105 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:56.105386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:04:56.105629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105352 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:56.105629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105551 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:56.105629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.105565 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:56.106408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.106380 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.106526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.106413 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.106526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.106429 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.106526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.106442 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.106526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.106456 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.107225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.107198 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:56.107313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.107227 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:04:56.107313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.107258 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "kube-api-access-pd5w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:04:56.107498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.107477 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy" pod "5f5963eb-38cb-45a2-9ae8-7e8a60d398da" (UID: "5f5963eb-38cb-45a2-9ae8-7e8a60d398da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:04:56.208436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.208402 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.208436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.208434 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pd5w9\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-kube-api-access-pd5w9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.208653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.208447 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.208653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.208461 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f5963eb-38cb-45a2-9ae8-7e8a60d398da-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:04:56.387631 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:56.387540 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:04:56.833361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.833321 6171 generic.go:356] "Generic (PLEG): container finished" podID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerID="6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e" exitCode=0 Oct 30 06:04:56.833581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.833382 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" Oct 30 06:04:56.833581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.833393 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerDied","Data":"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e"} Oct 30 06:04:56.833581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.833484 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j" event={"ID":"5f5963eb-38cb-45a2-9ae8-7e8a60d398da","Type":"ContainerDied","Data":"b3e2eda13380689d48aa75777bb9f04108263540658d6fec1bf7409805d531e3"} Oct 30 06:04:56.833581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.833513 6171 scope.go:122] "RemoveContainer" containerID="6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e" Oct 30 06:04:56.853730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.853707 6171 scope.go:122] "RemoveContainer" containerID="07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9" Oct 30 06:04:56.865084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.865010 6171 scope.go:122] "RemoveContainer" containerID="6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e" Oct 30 06:04:56.865398 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:56.865374 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e\": container with ID starting with 6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e not found: ID does not exist" containerID="6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e" Oct 30 06:04:56.865454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.865412 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e"} err="failed to get container status \"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e\": rpc error: code = NotFound desc = could not find container \"6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e\": container with ID starting with 6ea81eba1bfdfc7d19714776aa6ca944d49a14d56afeeb193e70261331c7780e not found: ID does not exist" Oct 30 06:04:56.865454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.865430 6171 scope.go:122] "RemoveContainer" containerID="07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9" Oct 30 06:04:56.865718 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:04:56.865699 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9\": container with ID starting with 07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9 not found: ID does not exist" containerID="07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9" Oct 30 06:04:56.865788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.865724 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9"} err="failed to get container status \"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9\": rpc error: code = NotFound desc = could not find container \"07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9\": container with ID starting with 07db51b8a8f7ca0daafaea44ab9473c05921bd82f8200d59e4d35429044796f9 not found: ID does not exist" Oct 30 06:04:56.910384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.910349 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:04:56.915524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:56.915492 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-f78ca5-openshift-default-d6b5dd7dc-scp8j"] Oct 30 06:04:57.501073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.494016 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:04:57.762116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.755465 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" path="/var/lib/kubelet/pods/5f5963eb-38cb-45a2-9ae8-7e8a60d398da/volumes" Oct 30 06:04:57.880459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.880425 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:04:57.881907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.881883 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.882124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.882096 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.882273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.882216 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.882273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.882226 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.882489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.882471 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.882558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.882513 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5963eb-38cb-45a2-9ae8-7e8a60d398da" containerName="istio-proxy" Oct 30 06:04:57.888787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.888768 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:57.909385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.909360 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:04:57.913207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.913180 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7df9fa\"/\"openshift-service-ca.crt\"" Oct 30 06:04:57.913207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.913180 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7df9fa\"/\"test-model-7df9fa-kserve-self-signed-certs\"" Oct 30 06:04:57.913207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.913211 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7df9fa\"/\"default-dockercfg-zm7j2\"" Oct 30 06:04:57.913486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:57.913211 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7df9fa\"/\"kube-root-ca.crt\"" Oct 30 06:04:58.036112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.035991 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds"] Oct 30 06:04:58.040877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.040856 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae0f0b17-pntds"] Oct 30 06:04:58.050272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050247 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.050377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050282 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.050451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050431 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.050606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050588 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.050709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050695 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt8bp\" (UniqueName: \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.050755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.050744 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.152761 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.152943 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.153073 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt8bp\" (UniqueName: \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.153151 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.153282 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.153415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.153318 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.167081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.153932 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.167081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.154178 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.167081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.163448 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.188787 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt8bp\" (UniqueName: \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.188823 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.189129 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\") pod \"test-model-7df9fa-kserve-74c8d484df-m8f74\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.212332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.211406 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:04:58.422130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.411934 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:04:58.448315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.440560 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:58.448315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.446412 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:04:58.460943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.460840 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d3e7e6\"" Oct 30 06:04:58.461180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.461150 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-d3e7e6-dockercfg-wbbtg\"" Oct 30 06:04:58.658356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.640010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4zt8\" (UniqueName: \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:58.658356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.640144 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:58.774075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.743036 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4zt8\" (UniqueName: \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:58.774075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.743162 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:58.837110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:58.835601 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:59.074014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:59.073516 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4zt8\" (UniqueName: \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\") pod \"payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:59.367180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:59.367146 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:04:59.382162 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:59.382125 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb32ad1e0_cd7c_44fb_b35f_0f4785910771.slice/crio-69850e3c3a5417ed86da0c2f9bbc9803f3e41c8192020d16fdb02d7f0b908040 WatchSource:0}: Error finding container 69850e3c3a5417ed86da0c2f9bbc9803f3e41c8192020d16fdb02d7f0b908040: Status 404 returned error can't find the container with id 69850e3c3a5417ed86da0c2f9bbc9803f3e41c8192020d16fdb02d7f0b908040 Oct 30 06:04:59.383151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:59.383120 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:04:59.677769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:59.677688 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bef560e-5b00-4911-a2ce-460b83332f24" path="/var/lib/kubelet/pods/1bef560e-5b00-4911-a2ce-460b83332f24/volumes" Oct 30 06:04:59.696526 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:04:59.696492 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8df40470_788d_4c78_85e5_32c2b75535b8.slice/crio-b8c12a02f29654b0f1ced029e686bc648faeb296771789da6fed2f36df67b157 WatchSource:0}: Error finding container b8c12a02f29654b0f1ced029e686bc648faeb296771789da6fed2f36df67b157: Status 404 returned error can't find the container with id b8c12a02f29654b0f1ced029e686bc648faeb296771789da6fed2f36df67b157 Oct 30 06:04:59.698110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:04:59.698084 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:05:00.001233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:00.001180 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:05:00.956076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:00.946805 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:05:01.399072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.397702 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.444088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.429430 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" event={"ID":"b32ad1e0-cd7c-44fb-b35f-0f4785910771","Type":"ContainerStarted","Data":"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0"} Oct 30 06:05:01.444088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.429505 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" event={"ID":"b32ad1e0-cd7c-44fb-b35f-0f4785910771","Type":"ContainerStarted","Data":"69850e3c3a5417ed86da0c2f9bbc9803f3e41c8192020d16fdb02d7f0b908040"} Oct 30 06:05:01.474116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.471916 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.499074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.494732 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.547099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.540244 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podStartSLOduration=4.540222035 podStartE2EDuration="4.540222035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:01.50955151 +0000 UTC m=+1959.949928706" watchObservedRunningTime="2026-10-30 06:05:01.540222035 +0000 UTC m=+1959.980599230" Oct 30 06:05:01.564075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.555468 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" event={"ID":"8df40470-788d-4c78-85e5-32c2b75535b8","Type":"ContainerStarted","Data":"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db"} Oct 30 06:05:01.588237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.588200 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:05:01.588566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.588538 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" event={"ID":"8df40470-788d-4c78-85e5-32c2b75535b8","Type":"ContainerStarted","Data":"b8c12a02f29654b0f1ced029e686bc648faeb296771789da6fed2f36df67b157"} Oct 30 06:05:01.623075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.617587 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" podStartSLOduration=3.617564315 podStartE2EDuration="3.617564315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:01.616541538 +0000 UTC m=+1960.056918732" watchObservedRunningTime="2026-10-30 06:05:01.617564315 +0000 UTC m=+1960.057941510" Oct 30 06:05:01.757938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.742866 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:05:01.848072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.809219 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.905071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.893199 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.905071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.894874 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.905071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.894981 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:01.905071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:01.895026 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:05:02.023355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.012766 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.313207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.294999 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/health\": context deadline exceeded" Oct 30 06:05:02.360102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.327917 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:05:02.360102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.328152 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.360102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.330404 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:05:02.360102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.330437 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 06:05:02.360102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.342211 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.599198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.599097 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.599851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.599828 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:05:02.764079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.759582 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.764079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.760202 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:05:02.764079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.762600 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/health\": context deadline exceeded" Oct 30 06:05:02.908766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.901349 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:02.908766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:02.901879 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:05:03.120526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.120116 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:05:03.133263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.133184 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:05:03.185159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.167864 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:05:03.239164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.234585 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": context deadline exceeded" Oct 30 06:05:03.239164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.235217 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:05:03.239164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.235400 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:05:03.239164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.236587 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:05:03.254062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.249949 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": context deadline exceeded" Oct 30 06:05:03.274074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.274025 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:05:03.292204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.292168 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:05:03.332561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:03.332531 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:05:08.218108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:08.218022 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:05:08.231825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:08.231799 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:05:08.794280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:08.794250 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:05:12.641886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.641801 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:05:12.826319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.826287 6171 generic.go:356] "Generic (PLEG): container finished" podID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerID="84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153" exitCode=0 Oct 30 06:05:12.826501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.826367 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerDied","Data":"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153"} Oct 30 06:05:12.826968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.826950 6171 scope.go:122] "RemoveContainer" containerID="84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153" Oct 30 06:05:12.862309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.862254 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:12.862632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.862605 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:12.863138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:12.863108 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:13.838544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:13.838512 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerStarted","Data":"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9"} Oct 30 06:05:14.722920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:14.722887 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:05:14.723173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:14.723061 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:14.723173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:14.723126 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:15.723162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:15.723125 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:15.723584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:15.723188 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:16.723206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.723168 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:16.723637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.723226 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:16.891801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.891764 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:05:16.899029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.899006 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.922270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.922237 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:05:16.926955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.926925 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7ad9ad-openshift-default-dockercfg-kg24d\"" Oct 30 06:05:16.976363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976269 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5ljk\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976389 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976442 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976550 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976667 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976711 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976810 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.976945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.976930 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:16.977023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:16.977010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078431 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078524 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078607 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078644 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078693 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078794 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.078876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078861 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078916 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5ljk\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.078965 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.079029 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.079088 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.079287 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.079381 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.079557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.079538 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.104092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.103994 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.107360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.107276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.108493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.108462 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5ljk\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.108493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.108480 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\") pod \"e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.227570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.227488 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:17.611671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.611641 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:05:17.612750 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:05:17.612716 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7913728c_2dbb_4b30_bf7e_ae6c70211426.slice/crio-0baf19237e8e1814cfbb848422af163abc8e7d11dd0c55c2917ce77893e5ab19 WatchSource:0}: Error finding container 0baf19237e8e1814cfbb848422af163abc8e7d11dd0c55c2917ce77893e5ab19: Status 404 returned error can't find the container with id 0baf19237e8e1814cfbb848422af163abc8e7d11dd0c55c2917ce77893e5ab19 Oct 30 06:05:17.615857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.615822 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:17.615966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.615895 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:17.615966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.615939 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:05:17.846068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.836210 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:17.846068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.836301 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:17.919608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:17.919567 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" event={"ID":"7913728c-2dbb-4b30-bf7e-ae6c70211426","Type":"ContainerStarted","Data":"0baf19237e8e1814cfbb848422af163abc8e7d11dd0c55c2917ce77893e5ab19"} Oct 30 06:05:18.732072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:18.723527 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:18.732072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:18.723609 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:18.940322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:18.940292 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" event={"ID":"7913728c-2dbb-4b30-bf7e-ae6c70211426","Type":"ContainerStarted","Data":"6a9c3f5a5f9016b06e4766b1c51419f8b160766a4534d529eb53784b1d3d090d"} Oct 30 06:05:19.042120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.041158 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podStartSLOduration=3.041136927 podStartE2EDuration="3.041136927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:19.027451945 +0000 UTC m=+1977.467829140" watchObservedRunningTime="2026-10-30 06:05:19.041136927 +0000 UTC m=+1977.481514121" Oct 30 06:05:19.228392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.228349 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:19.230660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.230634 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:19.230855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.230834 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:19.722888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.722852 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" start-of-body= Oct 30 06:05:19.723103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:19.722921 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.254:15021/healthz/ready\": dial tcp 10.133.0.254:15021: connect: connection refused" Oct 30 06:05:20.228292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:20.228252 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:20.228802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:20.228312 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:20.728852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:20.728822 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:05:20.729355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:20.729342 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:05:20.730122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:20.730096 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:05:21.228010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:21.227982 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:21.228216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:21.228084 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:22.228525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:22.228494 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:22.229021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:22.228995 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:23.228692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:23.228658 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:23.229165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:23.228715 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:24.253069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:24.243199 6171 patch_prober.go:29] interesting pod/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" start-of-body= Oct 30 06:05:24.253069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:24.243267 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.15:15021/healthz/ready\": dial tcp 10.133.1.15:15021: connect: connection refused" Oct 30 06:05:25.233706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:25.233671 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:25.234226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:25.234206 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:25.234889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:25.234870 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:05:26.389226 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:05:26.389194 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:05:26.475980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:26.475948 6171 scope.go:122] "RemoveContainer" containerID="3b99719f3da86f9fd2eba03443dfdde47b85b5bf73dfda28f6052d7df7f8306d" Oct 30 06:05:35.527840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:35.527759 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 06:05:35.528322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:35.527956 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" containerID="cri-o://93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4" gracePeriod=30 Oct 30 06:05:35.542311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:35.542282 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 06:05:35.542482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:35.542461 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" containerID="cri-o://35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219" gracePeriod=30 Oct 30 06:05:36.612990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:36.612940 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-96dmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:36.613549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:36.613025 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:40.901167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.899377 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 06:05:40.946925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.946886 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.946990 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947022 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947127 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947214 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947423 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947456 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947616 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.947752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.947735 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\") pod \"21930d30-b311-42e1-bdeb-a82bd367255b\" (UID: \"21930d30-b311-42e1-bdeb-a82bd367255b\") " Oct 30 06:05:40.948962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.948929 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:40.953642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.953614 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:40.953949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.953925 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:40.954229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.954206 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:40.954988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.954966 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "kube-api-access-rmw68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:40.955204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.955183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:40.957674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.957647 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:40.957759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.957656 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:40.958541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.958503 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo" pod "21930d30-b311-42e1-bdeb-a82bd367255b" (UID: "21930d30-b311-42e1-bdeb-a82bd367255b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:40.972452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:40.972432 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 06:05:41.049587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049552 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.049773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.049849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049797 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.049849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049834 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.049961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049932 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.050019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.049963 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.050019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.050000 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.050205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.050063 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.050269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.050255 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\") pod \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\" (UID: \"51093a2f-5b98-4388-b1ba-834f8b5c4b1f\") " Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051015 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051077 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/21930d30-b311-42e1-bdeb-a82bd367255b-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051094 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051108 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmw68\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-kube-api-access-rmw68\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051122 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/21930d30-b311-42e1-bdeb-a82bd367255b-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051136 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051148 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051162 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/21930d30-b311-42e1-bdeb-a82bd367255b-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051175 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/21930d30-b311-42e1-bdeb-a82bd367255b-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.051751 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.052140 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.052450 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.052517 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:41.054242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.052711 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:41.055067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.054639 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:41.055067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.054769 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "kube-api-access-l2ztc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:41.056170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.056145 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:41.056875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.056855 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy" pod "51093a2f-5b98-4388-b1ba-834f8b5c4b1f" (UID: "51093a2f-5b98-4388-b1ba-834f8b5c4b1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169532 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169569 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2ztc\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-kube-api-access-l2ztc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169584 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169598 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169610 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169621 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169633 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169645 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.170125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.169658 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51093a2f-5b98-4388-b1ba-834f8b5c4b1f-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:05:41.328826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.328788 6171 generic.go:356] "Generic (PLEG): container finished" podID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerID="35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219" exitCode=0 Oct 30 06:05:41.329020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.328830 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" Oct 30 06:05:41.329020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.328868 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" event={"ID":"51093a2f-5b98-4388-b1ba-834f8b5c4b1f","Type":"ContainerDied","Data":"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219"} Oct 30 06:05:41.329020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.328931 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4" event={"ID":"51093a2f-5b98-4388-b1ba-834f8b5c4b1f","Type":"ContainerDied","Data":"1d3e1aca8fddfc51fa6b5023113a22c3789229bc775e5f6eeb7e639a1224cfbb"} Oct 30 06:05:41.329020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.328963 6171 scope.go:122] "RemoveContainer" containerID="35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219" Oct 30 06:05:41.331501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.331476 6171 generic.go:356] "Generic (PLEG): container finished" podID="21930d30-b311-42e1-bdeb-a82bd367255b" containerID="93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4" exitCode=0 Oct 30 06:05:41.331641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.331536 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" Oct 30 06:05:41.331641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.331564 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerDied","Data":"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4"} Oct 30 06:05:41.331641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.331619 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr" event={"ID":"21930d30-b311-42e1-bdeb-a82bd367255b","Type":"ContainerDied","Data":"7881b0a21147e72e429048ec4058489234f994347fd2259e08d443693716650a"} Oct 30 06:05:41.343244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.343224 6171 scope.go:122] "RemoveContainer" containerID="35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219" Oct 30 06:05:41.343559 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:05:41.343540 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219\": container with ID starting with 35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219 not found: ID does not exist" containerID="35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219" Oct 30 06:05:41.343670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.343568 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219"} err="failed to get container status \"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219\": rpc error: code = NotFound desc = could not find container \"35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219\": container with ID starting with 35327c9f559ad76975495686a09bfe7c4980b73402d2503529e9daca01088219 not found: ID does not exist" Oct 30 06:05:41.343670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.343589 6171 scope.go:122] "RemoveContainer" containerID="93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4" Oct 30 06:05:41.368195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.368168 6171 scope.go:122] "RemoveContainer" containerID="764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab" Oct 30 06:05:41.452630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.451157 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 06:05:41.518361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.518331 6171 scope.go:122] "RemoveContainer" containerID="93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4" Oct 30 06:05:41.519480 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:05:41.519108 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4\": container with ID starting with 93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4 not found: ID does not exist" containerID="93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4" Oct 30 06:05:41.519480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.519194 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4"} err="failed to get container status \"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4\": rpc error: code = NotFound desc = could not find container \"93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4\": container with ID starting with 93111765af49da6c7ba5b8eaa46d6907c12d426a7031542b862e3ed0a1e193e4 not found: ID does not exist" Oct 30 06:05:41.519480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.519237 6171 scope.go:122] "RemoveContainer" containerID="764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab" Oct 30 06:05:41.519799 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:05:41.519553 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab\": container with ID starting with 764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab not found: ID does not exist" containerID="764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab" Oct 30 06:05:41.519799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.519582 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab"} err="failed to get container status \"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab\": rpc error: code = NotFound desc = could not find container \"764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab\": container with ID starting with 764ac6cc4bdcbd7f5280fde584c65f04ef858d3ac23c3533653f5f823788f0ab not found: ID does not exist" Oct 30 06:05:41.552074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.547426 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-96dmr"] Oct 30 06:05:41.569604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.569568 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 06:05:41.575771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.575709 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-2c9x4"] Oct 30 06:05:41.706811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.706725 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" path="/var/lib/kubelet/pods/21930d30-b311-42e1-bdeb-a82bd367255b/volumes" Oct 30 06:05:41.707310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:41.707290 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" path="/var/lib/kubelet/pods/51093a2f-5b98-4388-b1ba-834f8b5c4b1f/volumes" Oct 30 06:05:56.386631 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:05:56.386592 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:05:56.645125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:56.644199 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:05:59.800072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:59.791296 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:05:59.866067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:05:59.865600 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.271694 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.273248 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.255:8000/health\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.305146 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.305605 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.325799 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.325922 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.326655 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.330577 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.352299 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.354335 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:06:00.356137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.354723 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:06:00.382070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.361270 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.255:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.382070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.363268 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:00.662130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:00.646282 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:06:24.229818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.229787 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw"] Oct 30 06:06:24.230824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.230804 6171 cpu_manager.go:414] "containerMap: removing container" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:24.230893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.230826 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:24.230893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.230868 6171 cpu_manager.go:414] "containerMap: removing container" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" Oct 30 06:06:24.230893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.230874 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" Oct 30 06:06:24.231130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.231115 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="51093a2f-5b98-4388-b1ba-834f8b5c4b1f" containerName="istio-proxy" Oct 30 06:06:24.231177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.231164 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:24.231216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.231180 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:24.235743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.235721 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.256582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.256560 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw"] Oct 30 06:06:24.263583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.263556 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:24.263755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.263656 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:06:24.374003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.373970 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.374285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.374267 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql7xp\" (UniqueName: \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.374386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.374372 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.476225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.476192 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql7xp\" (UniqueName: \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.476391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.476290 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.476497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.476475 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.477213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.477102 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.478982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.478958 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.505143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.505088 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql7xp\" (UniqueName: \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\") pod \"maas-api-revoke-sub-db9e71ee-bxccw\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.553328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.553308 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:24.891271 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:06:24.891237 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc7dacb2_bb78_49f0_8fe0_a0ebd939eafc.slice/crio-adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e WatchSource:0}: Error finding container adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e: Status 404 returned error can't find the container with id adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e Oct 30 06:06:24.893964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:24.893936 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw"] Oct 30 06:06:25.743392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:25.743359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" event={"ID":"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc","Type":"ContainerStarted","Data":"adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e"} Oct 30 06:06:26.385090 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:06:26.385013 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:06:26.538486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:26.538407 6171 scope.go:122] "RemoveContainer" containerID="553a45841bc86163b107cbfa3ade2215958740ec3c161be7bf181060689744c4" Oct 30 06:06:26.742007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:26.741972 6171 generic.go:356] "Generic (PLEG): container finished" podID="bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" containerID="d7988cbced013c441840822c2e24cc8d0915769405975745587d4c9c647a7069" exitCode=0 Oct 30 06:06:26.742264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:26.742098 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" event={"ID":"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc","Type":"ContainerDied","Data":"d7988cbced013c441840822c2e24cc8d0915769405975745587d4c9c647a7069"} Oct 30 06:06:27.981592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:27.981566 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:28.142717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.142634 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\") pod \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " Oct 30 06:06:28.142866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.142759 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\") pod \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " Oct 30 06:06:28.142996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.142970 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\" (UniqueName: \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\") pod \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\" (UID: \"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc\") " Oct 30 06:06:28.143178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.143151 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca" pod "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" (UID: "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:28.143656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.143632 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:28.145147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.145123 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp" pod "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" (UID: "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc"). InnerVolumeSpecName "kube-api-access-ql7xp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:28.145256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.145220 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca" pod "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" (UID: "bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:28.245120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.245086 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:28.245120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.245120 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ql7xp\" (UniqueName: \"kubernetes.io/projected/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc-kube-api-access-ql7xp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:28.762792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.762759 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" Oct 30 06:06:28.762792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.762760 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw" event={"ID":"bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc","Type":"ContainerDied","Data":"adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e"} Oct 30 06:06:28.763115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:28.762815 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adcf83210ed526781d99a0b1810d19873a97e2c83b5c0b8a132daaf0b0a9f09e" Oct 30 06:06:29.380156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.380121 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:06:29.381680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.381654 6171 cpu_manager.go:414] "containerMap: removing container" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:29.381680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.381680 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21930d30-b311-42e1-bdeb-a82bd367255b" containerName="istio-proxy" Oct 30 06:06:29.381893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.381714 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" containerName="revoke-keys" Oct 30 06:06:29.381893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.381723 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" containerName="revoke-keys" Oct 30 06:06:29.382076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.382059 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" containerName="revoke-keys" Oct 30 06:06:29.388280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.388252 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.420407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.420381 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:06:29.426988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.426968 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0d0152-openshift-default-dockercfg-c9mvg\"" Oct 30 06:06:29.480230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.480193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.480424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.480245 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.480533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.480503 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.480616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.480583 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.480721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.480674 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.481227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.481198 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.481348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.481308 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.481426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.481348 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.481426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.481380 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.584464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.584421 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.585956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.585928 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.586301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.586278 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.586488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.586472 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.597066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.586649 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.597355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.597330 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.597585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.597564 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.597714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.597700 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.605071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.597811 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.605408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.587885 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.609064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.585233 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.616065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.602180 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.616256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.602732 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.616376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.602924 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.616481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.605911 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.655460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.655379 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.657734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.657704 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.703637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.694870 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\") pod \"e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:29.706641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:29.706610 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:30.339829 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:06:30.339802 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a9794fc_a2c6_4710_a6d9_643fcaaafb5d.slice/crio-75d978d3d387df4a3272b6e0db1b99742a14a723a504da2e346cc651d39fac6c WatchSource:0}: Error finding container 75d978d3d387df4a3272b6e0db1b99742a14a723a504da2e346cc651d39fac6c: Status 404 returned error can't find the container with id 75d978d3d387df4a3272b6e0db1b99742a14a723a504da2e346cc651d39fac6c Oct 30 06:06:30.342597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.342578 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:06:30.343686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.343654 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:30.343773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.343720 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:30.343773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.343748 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:30.992167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.992132 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerStarted","Data":"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56"} Oct 30 06:06:30.992618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:30.992191 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerStarted","Data":"75d978d3d387df4a3272b6e0db1b99742a14a723a504da2e346cc651d39fac6c"} Oct 30 06:06:31.069297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:31.069235 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podStartSLOduration=2.069220643 podStartE2EDuration="2.069220643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:31.054774955 +0000 UTC m=+2049.495152148" watchObservedRunningTime="2026-10-30 06:06:31.069220643 +0000 UTC m=+2049.509597834" Oct 30 06:06:31.707971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:31.707937 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:31.714712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:31.714686 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:32.004251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:32.004222 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:32.005179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:32.005160 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:06:34.611594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.611565 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:06:34.616700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.616677 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.653068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.652035 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-7df9fa-openshift-default-dockercfg-whvxb\"" Oct 30 06:06:34.684389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.684358 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:06:34.749282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749246 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749448 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7l9f\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749510 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749618 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749658 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749679 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749774 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749811 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.749883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.749839 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851277 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7l9f\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851365 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851463 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851511 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851541 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851639 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851739 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851883 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.851982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.851950 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.852184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.852101 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.852423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.852396 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.852547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.852431 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.852618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.852469 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.852748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.852723 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.854188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.854169 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.854449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.854432 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.877013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.876953 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7l9f\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.877448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.877427 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\") pod \"e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:34.933586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:34.933557 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:35.107431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:35.107397 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:06:35.480701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:35.480659 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:35.480853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:35.480737 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:35.480853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:35.480767 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:06:35.488522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:35.486591 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:06:36.094516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.092136 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l"] Oct 30 06:06:36.100433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.100409 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.104420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.104322 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l"] Oct 30 06:06:36.216204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.197299 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:36.216204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.197434 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:06:36.250124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.245449 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sbcg\" (UniqueName: \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.250124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.245507 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.250124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.245528 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.251720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.251686 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" event={"ID":"cfe3ebc4-95c4-43a2-8994-548791ba7cc9","Type":"ContainerStarted","Data":"f6829214cdd4484a25ede93ec4afe72a155160801cc18cd187865dbfc1afb8e5"} Oct 30 06:06:36.251845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.251750 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" event={"ID":"cfe3ebc4-95c4-43a2-8994-548791ba7cc9","Type":"ContainerStarted","Data":"57ccccc5be6645c8b9847169287badf8f31a379c7e08ee5e53e0d449b96d2801"} Oct 30 06:06:36.307085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.306997 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podStartSLOduration=2.306983734 podStartE2EDuration="2.306983734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:36.301806672 +0000 UTC m=+2054.742183866" watchObservedRunningTime="2026-10-30 06:06:36.306983734 +0000 UTC m=+2054.747360925" Oct 30 06:06:36.347719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.347647 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sbcg\" (UniqueName: \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.347878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.347729 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.347878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.347749 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.348529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.348505 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.350526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.350507 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.373732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.373700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sbcg\" (UniqueName: \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\") pod \"maas-api-revoke-sub-b0e3355f-rsv2l\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.444367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.444323 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:36.789399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.789271 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l"] Oct 30 06:06:36.933981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.933949 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:36.935031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.935012 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:36.935031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:36.935062 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:37.262449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:37.262418 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" event={"ID":"135faeae-e50d-4ad5-859e-e4671c6c7c02","Type":"ContainerStarted","Data":"e9383a241efbae819ee99a09cea177c1f64295f26dae330550da348591a29bb0"} Oct 30 06:06:37.934481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:37.934446 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:37.934662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:37.934500 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:38.273810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:38.273777 6171 generic.go:356] "Generic (PLEG): container finished" podID="135faeae-e50d-4ad5-859e-e4671c6c7c02" containerID="2c09cec5d77940711f370a098a6b0f863c5c925276c536df336c4584d1a2da65" exitCode=0 Oct 30 06:06:38.274414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:38.273865 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" event={"ID":"135faeae-e50d-4ad5-859e-e4671c6c7c02","Type":"ContainerDied","Data":"2c09cec5d77940711f370a098a6b0f863c5c925276c536df336c4584d1a2da65"} Oct 30 06:06:38.934553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:38.934515 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:38.934740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:38.934574 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:39.732877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.732846 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:39.818066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.817444 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca" pod "135faeae-e50d-4ad5-859e-e4671c6c7c02" (UID: "135faeae-e50d-4ad5-859e-e4671c6c7c02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:39.821149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.788945 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\") pod \"135faeae-e50d-4ad5-859e-e4671c6c7c02\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " Oct 30 06:06:39.821305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.821284 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\" (UniqueName: \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\") pod \"135faeae-e50d-4ad5-859e-e4671c6c7c02\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " Oct 30 06:06:39.821425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.821408 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\") pod \"135faeae-e50d-4ad5-859e-e4671c6c7c02\" (UID: \"135faeae-e50d-4ad5-859e-e4671c6c7c02\") " Oct 30 06:06:39.822599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.822567 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca" pod "135faeae-e50d-4ad5-859e-e4671c6c7c02" (UID: "135faeae-e50d-4ad5-859e-e4671c6c7c02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:39.822718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.822615 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:39.824270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.824245 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg" pod "135faeae-e50d-4ad5-859e-e4671c6c7c02" (UID: "135faeae-e50d-4ad5-859e-e4671c6c7c02"). InnerVolumeSpecName "kube-api-access-5sbcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:39.924921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.924824 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sbcg\" (UniqueName: \"kubernetes.io/projected/135faeae-e50d-4ad5-859e-e4671c6c7c02-kube-api-access-5sbcg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:39.924921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.924853 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/135faeae-e50d-4ad5-859e-e4671c6c7c02-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:39.934714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.934689 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:39.934834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:39.934770 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:40.325116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:40.325028 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" Oct 30 06:06:40.325116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:40.325077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l" event={"ID":"135faeae-e50d-4ad5-859e-e4671c6c7c02","Type":"ContainerDied","Data":"e9383a241efbae819ee99a09cea177c1f64295f26dae330550da348591a29bb0"} Oct 30 06:06:40.325407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:40.325166 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9383a241efbae819ee99a09cea177c1f64295f26dae330550da348591a29bb0" Oct 30 06:06:40.934869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:40.934837 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:40.935319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:40.934901 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:41.934237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:41.934200 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body= Oct 30 06:06:41.934421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:41.934256 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" Oct 30 06:06:42.939391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:42.939362 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:42.939802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:42.939749 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:42.940422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:42.940403 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:06:50.130680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.130624 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" containerID="cri-o://a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d" gracePeriod=45 Oct 30 06:06:50.453282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.453255 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:06:50.477580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.477555 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\" (UniqueName: \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.477735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.477657 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.477801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.477759 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\" (UniqueName: \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.477928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.477906 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.478085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.477955 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.478153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.478136 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\") pod \"d0471402-79d6-4cae-b25e-1a1745e1f69f\" (UID: \"d0471402-79d6-4cae-b25e-1a1745e1f69f\") " Oct 30 06:06:50.478212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.478188 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:50.478449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.478422 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:50.478512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.478460 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:50.479490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.479471 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:50.479557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.479499 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:50.479557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.479513 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:50.480114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.480093 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "kube-api-access-vrt5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:50.480203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.480111 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:50.480373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.480356 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs" pod "d0471402-79d6-4cae-b25e-1a1745e1f69f" (UID: "d0471402-79d6-4cae-b25e-1a1745e1f69f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:50.497783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.497742 6171 generic.go:356] "Generic (PLEG): container finished" podID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerID="a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d" exitCode=0 Oct 30 06:06:50.497889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.497797 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" Oct 30 06:06:50.497889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.497831 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" event={"ID":"d0471402-79d6-4cae-b25e-1a1745e1f69f","Type":"ContainerDied","Data":"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d"} Oct 30 06:06:50.497889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.497870 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" event={"ID":"d0471402-79d6-4cae-b25e-1a1745e1f69f","Type":"ContainerDied","Data":"43a35d771cec608e27447a5dcdd9948c5cbf31b80bb00daf1dd6afc31ab2ff4b"} Oct 30 06:06:50.497889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.497884 6171 scope.go:122] "RemoveContainer" containerID="a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d" Oct 30 06:06:50.524654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.524627 6171 scope.go:122] "RemoveContainer" containerID="a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d" Oct 30 06:06:50.525211 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:06:50.525187 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d\": container with ID starting with a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d not found: ID does not exist" containerID="a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d" Oct 30 06:06:50.525307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.525217 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d"} err="failed to get container status \"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d\": rpc error: code = NotFound desc = could not find container \"a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d\": container with ID starting with a335a6a073ee00eace69ef2da9a83b5b86ddf174644634b8643035dfe439ad6d not found: ID does not exist" Oct 30 06:06:50.570240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.570217 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:06:50.575559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.575538 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7"] Oct 30 06:06:50.581275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.581253 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d0471402-79d6-4cae-b25e-1a1745e1f69f-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:50.581275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.581273 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d0471402-79d6-4cae-b25e-1a1745e1f69f-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:50.581401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:50.581284 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrt5c\" (UniqueName: \"kubernetes.io/projected/d0471402-79d6-4cae-b25e-1a1745e1f69f-kube-api-access-vrt5c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:06:51.353905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:51.353872 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-f4e35c/test-model-f4e35c-kserve-75bfd44d7-lgzh7" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.255:8000/ready\": context deadline exceeded" Oct 30 06:06:51.677641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:06:51.677564 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" path="/var/lib/kubelet/pods/d0471402-79d6-4cae-b25e-1a1745e1f69f/volumes" Oct 30 06:06:56.382421 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:06:56.382379 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:07:04.810068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:04.805941 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:07:05.597365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.597331 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:07:05.907261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.907158 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq"] Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909289 6171 cpu_manager.go:414] "containerMap: removing container" podUID="135faeae-e50d-4ad5-859e-e4671c6c7c02" containerName="revoke-keys" Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909310 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="135faeae-e50d-4ad5-859e-e4671c6c7c02" containerName="revoke-keys" Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909352 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909357 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909518 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="135faeae-e50d-4ad5-859e-e4671c6c7c02" containerName="revoke-keys" Oct 30 06:07:05.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.909532 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0471402-79d6-4cae-b25e-1a1745e1f69f" containerName="main" Oct 30 06:07:05.916113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.916082 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq"] Oct 30 06:07:05.916280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.916163 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:05.933465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.933442 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:07:05.933626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.933551 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:07:05.986717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.986661 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:05.987102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.987029 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmp5l\" (UniqueName: \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:05.987236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:05.987120 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.088805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.088768 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmp5l\" (UniqueName: \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.088984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.088813 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.088984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.088919 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.089502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.089482 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.091471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.091440 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.115147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.115119 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmp5l\" (UniqueName: \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\") pod \"maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:06.235880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:06.235850 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:07.311071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.310550 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:07.342073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.316843 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:07.342073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.318622 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:07:07.381075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.380652 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:07.490917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.490873 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:07.958214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:07.958162 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.9:8000/ready\": context deadline exceeded" Oct 30 06:07:08.089073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.041010 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.125123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.123745 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.159073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.045112 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.214122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.148875 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.230117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.170529 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": context deadline exceeded" Oct 30 06:07:08.352265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.342865 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.379152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.348289 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:07:08.429394 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.348323 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": context deadline exceeded" Oct 30 06:07:08.508933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.348414 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.509447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.348437 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:07:08.510120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.351686 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.536480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.351740 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:08.536922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.170562 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": context deadline exceeded" Oct 30 06:07:08.537244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.390218 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:07:08.809433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.791631 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq"] Oct 30 06:07:08.809433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.791815 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:07:08.809433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.791858 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:07:08.809433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.791892 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:07:08.809433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.791928 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:07:08.934935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:08.910568 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" event={"ID":"970d6ecb-00a4-48ae-868a-12f6ddee9820","Type":"ContainerStarted","Data":"06398cd845c3a725563753d88fe7176c740a586ef9b2f1e6f7e7326c3976c22a"} Oct 30 06:07:09.079651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.075828 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.9:8000/ready\": context deadline exceeded" Oct 30 06:07:09.106071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.089522 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:07:09.125229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.125185 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.125859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.125833 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:07:09.264068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.231014 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:07:09.264068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.232870 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:07:09.264068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.254160 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.264068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.256135 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:07:09.373278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.366062 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" probeResult="failure" output="Get \"https://10.133.1.11:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.373278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.367455 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:07:09.431703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.385943 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": context deadline exceeded" Oct 30 06:07:09.431703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.387524 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:07:09.431703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.404364 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.472313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.434267 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:07:09.472313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.453624 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.518118 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.522077 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.534911 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.536663 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.550744 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.585518 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:09.653211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.586824 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:07:09.698562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.698533 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:07:09.877073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.871153 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": context deadline exceeded" Oct 30 06:07:09.877073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.871513 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": context deadline exceeded" Oct 30 06:07:09.877073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:09.871770 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": context deadline exceeded" Oct 30 06:07:10.111443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.111395 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.9:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:10.161169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.160577 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:07:10.161169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.160664 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:07:10.161169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.160922 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:07:10.161169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.160992 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:07:10.161169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:07:10.161577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161237 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:07:10.161577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161284 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:07:10.161577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161431 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:07:10.161577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161475 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:07:10.161577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161518 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:07:10.161843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161685 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:07:10.161843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161733 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:07:10.161843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161776 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:07:10.162074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161947 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:07:10.162074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.161996 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:07:10.239555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:10.239517 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:07:11.465618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:11.465524 6171 generic.go:356] "Generic (PLEG): container finished" podID="970d6ecb-00a4-48ae-868a-12f6ddee9820" containerID="1684dcaab9230b43925d2f95b85fd6d7f9d8bf403b9cfa25bd0e93bc438e9d2a" exitCode=0 Oct 30 06:07:11.466119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:11.466014 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" event={"ID":"970d6ecb-00a4-48ae-868a-12f6ddee9820","Type":"ContainerDied","Data":"1684dcaab9230b43925d2f95b85fd6d7f9d8bf403b9cfa25bd0e93bc438e9d2a"} Oct 30 06:07:12.055243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.055207 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2"] Oct 30 06:07:12.068977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.068934 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-13b4d176-846c2"] Oct 30 06:07:12.736224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.736158 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:12.860564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.860527 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\") pod \"970d6ecb-00a4-48ae-868a-12f6ddee9820\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " Oct 30 06:07:12.860761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.860573 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\") pod \"970d6ecb-00a4-48ae-868a-12f6ddee9820\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " Oct 30 06:07:12.860761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.860606 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\" (UniqueName: \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\") pod \"970d6ecb-00a4-48ae-868a-12f6ddee9820\" (UID: \"970d6ecb-00a4-48ae-868a-12f6ddee9820\") " Oct 30 06:07:12.860959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.860934 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca" pod "970d6ecb-00a4-48ae-868a-12f6ddee9820" (UID: "970d6ecb-00a4-48ae-868a-12f6ddee9820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:12.861371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.861352 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.862785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.862753 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca" pod "970d6ecb-00a4-48ae-868a-12f6ddee9820" (UID: "970d6ecb-00a4-48ae-868a-12f6ddee9820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:12.862916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.862898 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l" pod "970d6ecb-00a4-48ae-868a-12f6ddee9820" (UID: "970d6ecb-00a4-48ae-868a-12f6ddee9820"). InnerVolumeSpecName "kube-api-access-mmp5l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:12.962940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.962899 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/970d6ecb-00a4-48ae-868a-12f6ddee9820-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:12.962940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:12.962937 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmp5l\" (UniqueName: \"kubernetes.io/projected/970d6ecb-00a4-48ae-868a-12f6ddee9820-kube-api-access-mmp5l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:13.513673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:13.513636 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" Oct 30 06:07:13.513673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:13.513658 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq" event={"ID":"970d6ecb-00a4-48ae-868a-12f6ddee9820","Type":"ContainerDied","Data":"06398cd845c3a725563753d88fe7176c740a586ef9b2f1e6f7e7326c3976c22a"} Oct 30 06:07:13.513898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:13.513707 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06398cd845c3a725563753d88fe7176c740a586ef9b2f1e6f7e7326c3976c22a" Oct 30 06:07:13.676396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:13.676360 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0546563a-a40f-4e32-b234-2a43c2e51b66" path="/var/lib/kubelet/pods/0546563a-a40f-4e32-b234-2a43c2e51b66/volumes" Oct 30 06:07:16.354350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:16.354298 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 06:07:16.368075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:16.368023 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 06:07:20.593430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.593393 6171 generic.go:356] "Generic (PLEG): container finished" podID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerID="70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2" exitCode=0 Oct 30 06:07:20.593936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.593503 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerDied","Data":"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2"} Oct 30 06:07:20.594163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.594130 6171 scope.go:122] "RemoveContainer" containerID="70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2" Oct 30 06:07:20.596438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.596401 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:20.596552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.596485 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:20.596552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.596524 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:07:20.620189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.620159 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" containerID="cri-o://5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6" gracePeriod=45 Oct 30 06:07:20.835141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:20.827542 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" probeResult="failure" output="Get \"https://10.133.1.2:8000/ready\": dial tcp 10.133.1.2:8000: connect: connection refused" Oct 30 06:07:21.208294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.208266 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:07:21.356058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.355943 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" containerID="cri-o://68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab" gracePeriod=25 Oct 30 06:07:21.380347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.380289 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" containerID="cri-o://42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef" gracePeriod=25 Oct 30 06:07:21.407176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.400165 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\" (UniqueName: \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.407627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.407598 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\" (UniqueName: \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.469192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.452210 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.469192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.452276 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.469192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.452425 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.469192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.452630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\") pod \"af19d653-5b0a-4e26-9609-31d9065f59cc\" (UID: \"af19d653-5b0a-4e26-9609-31d9065f59cc\") " Oct 30 06:07:21.477356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.475415 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:21.477356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.475639 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "kube-api-access-n4t5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:21.477356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.476437 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:21.477356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.476668 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:21.483426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.478115 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:21.483426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.479098 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm" pod "af19d653-5b0a-4e26-9609-31d9065f59cc" (UID: "af19d653-5b0a-4e26-9609-31d9065f59cc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:21.555666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555634 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4t5d\" (UniqueName: \"kubernetes.io/projected/af19d653-5b0a-4e26-9609-31d9065f59cc-kube-api-access-n4t5d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.555666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555663 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/af19d653-5b0a-4e26-9609-31d9065f59cc-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.555666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555675 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.555909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555683 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.555909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555694 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.555909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.555701 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/af19d653-5b0a-4e26-9609-31d9065f59cc-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:21.606459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.606377 6171 generic.go:356] "Generic (PLEG): container finished" podID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerID="5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6" exitCode=0 Oct 30 06:07:21.606955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.606659 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" event={"ID":"af19d653-5b0a-4e26-9609-31d9065f59cc","Type":"ContainerDied","Data":"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6"} Oct 30 06:07:21.606955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.606688 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" Oct 30 06:07:21.606955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.606707 6171 scope.go:122] "RemoveContainer" containerID="5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6" Oct 30 06:07:21.606955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.606695 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf" event={"ID":"af19d653-5b0a-4e26-9609-31d9065f59cc","Type":"ContainerDied","Data":"13b43c0d6d99dd277f6d63208e8d035f10a3b0a59ee815b5a2525c050bc48918"} Oct 30 06:07:21.610763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.610727 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerStarted","Data":"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b"} Oct 30 06:07:21.622857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.622826 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:07:21.626102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.626076 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:21.626205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.626125 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:21.626565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.626534 6171 scope.go:122] "RemoveContainer" containerID="5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6" Oct 30 06:07:21.626942 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:21.626914 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6\": container with ID starting with 5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6 not found: ID does not exist" containerID="5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6" Oct 30 06:07:21.627028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.626953 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6"} err="failed to get container status \"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6\": rpc error: code = NotFound desc = could not find container \"5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6\": container with ID starting with 5b659af571c853ab8fce8943a5d93a3790660699fd4462d3fcdb99cfe5937cf6 not found: ID does not exist" Oct 30 06:07:21.724891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.724850 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:07:21.730258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:21.730230 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-3cc570/test-model-3cc570-kserve-79d9685cc-jckvf"] Oct 30 06:07:22.624386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:22.624264 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:22.624386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:22.624322 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:23.623389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:23.623348 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:23.623588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:23.623432 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:23.687156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:23.680180 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" path="/var/lib/kubelet/pods/af19d653-5b0a-4e26-9609-31d9065f59cc/volumes" Oct 30 06:07:24.623203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:24.623167 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:24.623786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:24.623749 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:24.750086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:24.749928 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:07:25.624380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:25.624335 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:25.624609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:25.624405 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:26.155964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:26.155927 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv"] Oct 30 06:07:26.173081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:26.173022 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edde6690-qj8wv"] Oct 30 06:07:26.461713 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:26.461586 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:07:26.624475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:26.624142 6171 patch_prober.go:29] interesting pod/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" start-of-body= Oct 30 06:07:26.624475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:26.624185 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.12:15021/healthz/ready\": dial tcp 10.133.1.12:15021: connect: connection refused" Oct 30 06:07:27.629508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:27.629473 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:07:27.630426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:27.630403 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:07:27.674908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:27.674869 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="601ada4f-cb28-4958-8ddb-4e8974621dd0" path="/var/lib/kubelet/pods/601ada4f-cb28-4958-8ddb-4e8974621dd0/volumes" Oct 30 06:07:28.230065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:28.228380 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:28.236061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:28.232436 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:31.626637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.626595 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn"] Oct 30 06:07:31.627770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627741 6171 cpu_manager.go:414] "containerMap: removing container" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" Oct 30 06:07:31.627770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627771 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" Oct 30 06:07:31.627881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627783 6171 cpu_manager.go:414] "containerMap: removing container" podUID="970d6ecb-00a4-48ae-868a-12f6ddee9820" containerName="revoke-keys" Oct 30 06:07:31.627881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627788 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="970d6ecb-00a4-48ae-868a-12f6ddee9820" containerName="revoke-keys" Oct 30 06:07:31.627948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627940 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="af19d653-5b0a-4e26-9609-31d9065f59cc" containerName="main" Oct 30 06:07:31.627948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.627947 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="970d6ecb-00a4-48ae-868a-12f6ddee9820" containerName="revoke-keys" Oct 30 06:07:31.633865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.633845 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.652657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.652630 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn"] Oct 30 06:07:31.658618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.658589 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:07:31.658808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.658647 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:07:31.739592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.739555 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc4g4\" (UniqueName: \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.739768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.739623 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.739886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.739862 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.841671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.841637 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc4g4\" (UniqueName: \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.841876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.841699 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.841956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.841892 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.842503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.842482 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.844265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.844242 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.867119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.867094 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc4g4\" (UniqueName: \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\") pod \"maas-api-revoke-sub-bb936e39-kgpvn\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.951899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.951833 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:31.952751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.952728 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg"] Oct 30 06:07:31.958016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.957994 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:31.993963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:31.993931 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg"] Oct 30 06:07:32.046945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.045716 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsldw\" (UniqueName: \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.046945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.045997 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.046945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.046034 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.148281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.148242 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsldw\" (UniqueName: \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.148497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.148485 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.148593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.148517 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.149460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.149423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.151971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.151947 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.176556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.176523 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsldw\" (UniqueName: \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\") pod \"maas-api-revoke-sub-88df6884-pqffg\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.283632 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:07:32.283599 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod108f5f78_40c4_4fe9_bbd3_7b249c998f23.slice/crio-35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f WatchSource:0}: Error finding container 35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f: Status 404 returned error can't find the container with id 35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f Oct 30 06:07:32.284321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.284253 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn"] Oct 30 06:07:32.321418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.321393 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:32.644779 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:07:32.644748 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf21affb_a65e_40cd_be69_9833a5a06a92.slice/crio-94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17 WatchSource:0}: Error finding container 94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17: Status 404 returned error can't find the container with id 94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17 Oct 30 06:07:32.646594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.646564 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg"] Oct 30 06:07:32.734105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.734067 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" event={"ID":"bf21affb-a65e-40cd-be69-9833a5a06a92","Type":"ContainerStarted","Data":"94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17"} Oct 30 06:07:32.735583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:32.735557 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" event={"ID":"108f5f78-40c4-4fe9-bbd3-7b249c998f23","Type":"ContainerStarted","Data":"35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f"} Oct 30 06:07:33.747531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:33.747492 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" event={"ID":"bf21affb-a65e-40cd-be69-9833a5a06a92","Type":"ContainerStarted","Data":"c58ddc3db0340b3089e6d4b38358730547876b4272199168deda783fbdf5004a"} Oct 30 06:07:33.749400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:33.749371 6171 generic.go:356] "Generic (PLEG): container finished" podID="108f5f78-40c4-4fe9-bbd3-7b249c998f23" containerID="9d4ca657cc22109314b5f6b13013fd44926dbda88e0a67784f62a5fc49595547" exitCode=0 Oct 30 06:07:33.749538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:33.749436 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" event={"ID":"108f5f78-40c4-4fe9-bbd3-7b249c998f23","Type":"ContainerDied","Data":"9d4ca657cc22109314b5f6b13013fd44926dbda88e0a67784f62a5fc49595547"} Oct 30 06:07:33.797992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:33.797939 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" podStartSLOduration=2.79792443 podStartE2EDuration="2.79792443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:33.792743347 +0000 UTC m=+2112.233120539" watchObservedRunningTime="2026-10-30 06:07:33.79792443 +0000 UTC m=+2112.238301622" Oct 30 06:07:34.777876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:34.777798 6171 generic.go:356] "Generic (PLEG): container finished" podID="bf21affb-a65e-40cd-be69-9833a5a06a92" containerID="c58ddc3db0340b3089e6d4b38358730547876b4272199168deda783fbdf5004a" exitCode=0 Oct 30 06:07:34.778479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:34.777887 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" event={"ID":"bf21affb-a65e-40cd-be69-9833a5a06a92","Type":"ContainerDied","Data":"c58ddc3db0340b3089e6d4b38358730547876b4272199168deda783fbdf5004a"} Oct 30 06:07:34.987309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:34.987281 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:35.105010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.104913 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\") pod \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " Oct 30 06:07:35.105195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.105175 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\") pod \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " Oct 30 06:07:35.105253 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.105223 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\" (UniqueName: \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\") pod \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\" (UID: \"108f5f78-40c4-4fe9-bbd3-7b249c998f23\") " Oct 30 06:07:35.105370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.105342 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca" pod "108f5f78-40c4-4fe9-bbd3-7b249c998f23" (UID: "108f5f78-40c4-4fe9-bbd3-7b249c998f23"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:35.106334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.106307 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:35.108196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.108170 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca" pod "108f5f78-40c4-4fe9-bbd3-7b249c998f23" (UID: "108f5f78-40c4-4fe9-bbd3-7b249c998f23"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:35.108366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.108224 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4" pod "108f5f78-40c4-4fe9-bbd3-7b249c998f23" (UID: "108f5f78-40c4-4fe9-bbd3-7b249c998f23"). InnerVolumeSpecName "kube-api-access-tc4g4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:35.208527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.208503 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/108f5f78-40c4-4fe9-bbd3-7b249c998f23-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:35.208527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.208526 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc4g4\" (UniqueName: \"kubernetes.io/projected/108f5f78-40c4-4fe9-bbd3-7b249c998f23-kube-api-access-tc4g4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:35.788681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.788641 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" event={"ID":"108f5f78-40c4-4fe9-bbd3-7b249c998f23","Type":"ContainerDied","Data":"35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f"} Oct 30 06:07:35.789103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.788704 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35a8991d9137bf739145d1863a9fb05a9411fafcb0b006589c8eccca246f7f9f" Oct 30 06:07:35.789103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.788665 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn" Oct 30 06:07:35.973805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:35.973779 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:36.131452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.131425 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\" (UniqueName: \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\") pod \"bf21affb-a65e-40cd-be69-9833a5a06a92\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " Oct 30 06:07:36.131670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.131549 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\") pod \"bf21affb-a65e-40cd-be69-9833a5a06a92\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " Oct 30 06:07:36.131670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.131657 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\") pod \"bf21affb-a65e-40cd-be69-9833a5a06a92\" (UID: \"bf21affb-a65e-40cd-be69-9833a5a06a92\") " Oct 30 06:07:36.132008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.131979 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca" pod "bf21affb-a65e-40cd-be69-9833a5a06a92" (UID: "bf21affb-a65e-40cd-be69-9833a5a06a92"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:36.132699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.132672 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:36.133943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.133917 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw" pod "bf21affb-a65e-40cd-be69-9833a5a06a92" (UID: "bf21affb-a65e-40cd-be69-9833a5a06a92"). InnerVolumeSpecName "kube-api-access-vsldw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:36.134204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.134175 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca" pod "bf21affb-a65e-40cd-be69-9833a5a06a92" (UID: "bf21affb-a65e-40cd-be69-9833a5a06a92"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:36.234272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.234242 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsldw\" (UniqueName: \"kubernetes.io/projected/bf21affb-a65e-40cd-be69-9833a5a06a92-kube-api-access-vsldw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:36.234272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.234272 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf21affb-a65e-40cd-be69-9833a5a06a92-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:36.801948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.801920 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" Oct 30 06:07:36.802486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.801947 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg" event={"ID":"bf21affb-a65e-40cd-be69-9833a5a06a92","Type":"ContainerDied","Data":"94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17"} Oct 30 06:07:36.802486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:36.801994 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94447d2fe54ff81b3932ebcbd21e9813be43ba9c0f6ee775da6ba4dc3185cd17" Oct 30 06:07:38.223483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:38.223445 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:38.223887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:38.223445 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:07:41.740240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.740202 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 06:07:41.744262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.744239 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 06:07:41.750915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.750895 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\" (UniqueName: \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\") pod \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " Oct 30 06:07:41.751037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.750965 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\") pod \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\" (UID: \"c5e04b67-ef5c-4b28-b524-ddd5314edd85\") " Oct 30 06:07:41.751315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.751294 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume" pod "c5e04b67-ef5c-4b28-b524-ddd5314edd85" (UID: "c5e04b67-ef5c-4b28-b524-ddd5314edd85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:41.751769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.751752 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c5e04b67-ef5c-4b28-b524-ddd5314edd85-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:41.753230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.753208 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc" pod "c5e04b67-ef5c-4b28-b524-ddd5314edd85" (UID: "c5e04b67-ef5c-4b28-b524-ddd5314edd85"). InnerVolumeSpecName "kube-api-access-jjtjc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:41.852482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.852447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\") pod \"87e94acb-b40c-4c28-bede-9c24108a5d99\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " Oct 30 06:07:41.852691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.852516 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\" (UniqueName: \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\") pod \"87e94acb-b40c-4c28-bede-9c24108a5d99\" (UID: \"87e94acb-b40c-4c28-bede-9c24108a5d99\") " Oct 30 06:07:41.852874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.852847 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume" pod "87e94acb-b40c-4c28-bede-9c24108a5d99" (UID: "87e94acb-b40c-4c28-bede-9c24108a5d99"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:41.854008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.853979 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e94acb-b40c-4c28-bede-9c24108a5d99-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:41.854213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.854013 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjtjc\" (UniqueName: \"kubernetes.io/projected/c5e04b67-ef5c-4b28-b524-ddd5314edd85-kube-api-access-jjtjc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:41.856776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.856745 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d" pod "87e94acb-b40c-4c28-bede-9c24108a5d99" (UID: "87e94acb-b40c-4c28-bede-9c24108a5d99"). InnerVolumeSpecName "kube-api-access-sdh8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:41.859743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.859713 6171 generic.go:356] "Generic (PLEG): container finished" podID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerID="42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef" exitCode=0 Oct 30 06:07:41.859857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.859764 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" Oct 30 06:07:41.859857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.859788 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" event={"ID":"c5e04b67-ef5c-4b28-b524-ddd5314edd85","Type":"ContainerDied","Data":"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef"} Oct 30 06:07:41.859857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.859838 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b" event={"ID":"c5e04b67-ef5c-4b28-b524-ddd5314edd85","Type":"ContainerDied","Data":"69d9e08a6e4085db3a16eac0b5740ca65b441e3267197da0bfbae30a12e060bf"} Oct 30 06:07:41.860037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.859869 6171 scope.go:122] "RemoveContainer" containerID="42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef" Oct 30 06:07:41.861801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.861776 6171 generic.go:356] "Generic (PLEG): container finished" podID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerID="68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab" exitCode=0 Oct 30 06:07:41.861895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.861831 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" Oct 30 06:07:41.861895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.861867 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" event={"ID":"87e94acb-b40c-4c28-bede-9c24108a5d99","Type":"ContainerDied","Data":"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab"} Oct 30 06:07:41.862037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.861919 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm" event={"ID":"87e94acb-b40c-4c28-bede-9c24108a5d99","Type":"ContainerDied","Data":"9efaaa214a1bde3c885ac22e3a5068ed7d417c201eac2ca23cf4adda63c36509"} Oct 30 06:07:41.875187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.875019 6171 scope.go:122] "RemoveContainer" containerID="42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef" Oct 30 06:07:41.875596 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:41.875570 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef\": container with ID starting with 42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef not found: ID does not exist" containerID="42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef" Oct 30 06:07:41.875741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.875606 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef"} err="failed to get container status \"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef\": rpc error: code = NotFound desc = could not find container \"42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef\": container with ID starting with 42d661b476497508460d90bc4c3853383d0ab370ea46a362f0b4d3d0269004ef not found: ID does not exist" Oct 30 06:07:41.875741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.875627 6171 scope.go:122] "RemoveContainer" containerID="68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab" Oct 30 06:07:41.889835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.889814 6171 scope.go:122] "RemoveContainer" containerID="68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab" Oct 30 06:07:41.890205 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:41.890181 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab\": container with ID starting with 68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab not found: ID does not exist" containerID="68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab" Oct 30 06:07:41.890294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.890216 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab"} err="failed to get container status \"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab\": rpc error: code = NotFound desc = could not find container \"68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab\": container with ID starting with 68dd1d58174d15b07f5a721af9de406024403a327dc2e68e7c35b20fd63b84ab not found: ID does not exist" Oct 30 06:07:41.921528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.921456 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 06:07:41.926733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.926711 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-f4e35c-66ff966c5c-n66dm"] Oct 30 06:07:41.955940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.955911 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdh8d\" (UniqueName: \"kubernetes.io/projected/87e94acb-b40c-4c28-bede-9c24108a5d99-kube-api-access-sdh8d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:41.999072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:41.995687 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 06:07:42.032064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:42.032008 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f4e35c-c74d84667-k5x2b"] Oct 30 06:07:43.677301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:43.677266 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" path="/var/lib/kubelet/pods/87e94acb-b40c-4c28-bede-9c24108a5d99/volumes" Oct 30 06:07:43.677874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:43.677858 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" path="/var/lib/kubelet/pods/c5e04b67-ef5c-4b28-b524-ddd5314edd85/volumes" Oct 30 06:07:44.850944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:44.850908 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:07:45.078163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.077111 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86"] Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078648 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078676 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078725 6171 cpu_manager.go:414] "containerMap: removing container" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078734 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078787 6171 cpu_manager.go:414] "containerMap: removing container" podUID="108f5f78-40c4-4fe9-bbd3-7b249c998f23" containerName="revoke-keys" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078796 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108f5f78-40c4-4fe9-bbd3-7b249c998f23" containerName="revoke-keys" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078843 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bf21affb-a65e-40cd-be69-9833a5a06a92" containerName="revoke-keys" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.078851 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf21affb-a65e-40cd-be69-9833a5a06a92" containerName="revoke-keys" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.079256 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf21affb-a65e-40cd-be69-9833a5a06a92" containerName="revoke-keys" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.079285 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5e04b67-ef5c-4b28-b524-ddd5314edd85" containerName="payload-pre-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.079305 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e94acb-b40c-4c28-bede-9c24108a5d99" containerName="payload-processing" Oct 30 06:07:45.082071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.079326 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="108f5f78-40c4-4fe9-bbd3-7b249c998f23" containerName="revoke-keys" Oct 30 06:07:45.217449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.217373 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.233101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.233072 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86"] Oct 30 06:07:45.236663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.236638 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:07:45.237028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.236981 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:07:45.304679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.304642 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.304913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.304892 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.305148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.305126 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncfxq\" (UniqueName: \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.406330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.406296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncfxq\" (UniqueName: \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.406524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.406388 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.406574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.406530 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.407125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.407103 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.409062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.409023 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.431923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.431899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncfxq\" (UniqueName: \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\") pod \"maas-api-revoke-sub-37ae9cd8-zlt86\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.459271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.459237 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn"] Oct 30 06:07:45.464559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.464539 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.486005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.485970 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn"] Oct 30 06:07:45.649069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.618180 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.654547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.649685 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.654965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.654930 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kskgm\" (UniqueName: \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.655226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.644327 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:45.767131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.766323 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.767131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.766648 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.767131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.766764 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kskgm\" (UniqueName: \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.772064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.770927 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.780978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.780943 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.797626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:45.797588 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kskgm\" (UniqueName: \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\") pod \"maas-api-revoke-sub-7b24fc33-j79bn\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:45.999159 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:07:45.999126 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539e68d2_4ece_47e7_aed6_4e727d3173be.slice/crio-e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef WatchSource:0}: Error finding container e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef: Status 404 returned error can't find the container with id e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef Oct 30 06:07:46.000329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.000305 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86"] Oct 30 06:07:46.088280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.088252 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:46.646154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.646118 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn"] Oct 30 06:07:46.930119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.930077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" event={"ID":"9f9187ee-2869-426d-9aa3-230ff2d45fee","Type":"ContainerStarted","Data":"be81173031b3c9cb2b50fc861bc6c8b192ffe75f05d8b1539948efc327a0e665"} Oct 30 06:07:46.932129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.932097 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" event={"ID":"539e68d2-4ece-47e7-aed6-4e727d3173be","Type":"ContainerStarted","Data":"fc5ad46bae00ce68b9be32c16a99c1b2f45ef9842f51c4e0de9a5cb74fd3a406"} Oct 30 06:07:46.932269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.932151 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" event={"ID":"539e68d2-4ece-47e7-aed6-4e727d3173be","Type":"ContainerStarted","Data":"e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef"} Oct 30 06:07:46.981161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:46.981099 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" podStartSLOduration=2.981083458 podStartE2EDuration="2.981083458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:46.979613032 +0000 UTC m=+2125.419990226" watchObservedRunningTime="2026-10-30 06:07:46.981083458 +0000 UTC m=+2125.421460648" Oct 30 06:07:47.944144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:47.944111 6171 generic.go:356] "Generic (PLEG): container finished" podID="9f9187ee-2869-426d-9aa3-230ff2d45fee" containerID="d2578e683b6fb8deee52ef7470fa1c30cba773557eb01dd4cb1e845295bc8b0c" exitCode=0 Oct 30 06:07:47.944613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:47.944198 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" event={"ID":"9f9187ee-2869-426d-9aa3-230ff2d45fee","Type":"ContainerDied","Data":"d2578e683b6fb8deee52ef7470fa1c30cba773557eb01dd4cb1e845295bc8b0c"} Oct 30 06:07:47.947101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:47.946028 6171 generic.go:356] "Generic (PLEG): container finished" podID="539e68d2-4ece-47e7-aed6-4e727d3173be" containerID="fc5ad46bae00ce68b9be32c16a99c1b2f45ef9842f51c4e0de9a5cb74fd3a406" exitCode=0 Oct 30 06:07:47.947101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:47.946135 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" event={"ID":"539e68d2-4ece-47e7-aed6-4e727d3173be","Type":"ContainerDied","Data":"fc5ad46bae00ce68b9be32c16a99c1b2f45ef9842f51c4e0de9a5cb74fd3a406"} Oct 30 06:07:49.240555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.239379 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/health\": context deadline exceeded" Oct 30 06:07:49.255467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.253152 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:49.290134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.288312 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:49.344184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.344133 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:49.472066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.466160 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 06:07:49.526680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.526634 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:49.609227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.609126 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:07:49.798068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.795204 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" probeResult="failure" output="Get \"https://10.133.1.6:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:49.883118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:49.879687 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:50.050341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.039546 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:50.137264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.137154 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" probeResult="failure" output="Get \"https://10.133.1.8:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:50.164070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.139546 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" probeResult="failure" output="Get \"https://10.133.1.9:8000/ready\": context deadline exceeded" Oct 30 06:07:50.206255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.170761 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:07:50.266235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.256125 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": context deadline exceeded" Oct 30 06:07:50.266235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.256712 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:07:50.351564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.348098 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:07:50.362518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.362489 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:50.419894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.419796 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\") pod \"539e68d2-4ece-47e7-aed6-4e727d3173be\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " Oct 30 06:07:50.420629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.420257 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\" (UniqueName: \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\") pod \"539e68d2-4ece-47e7-aed6-4e727d3173be\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " Oct 30 06:07:50.421105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.421084 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\") pod \"539e68d2-4ece-47e7-aed6-4e727d3173be\" (UID: \"539e68d2-4ece-47e7-aed6-4e727d3173be\") " Oct 30 06:07:50.422707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.422682 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca" pod "539e68d2-4ece-47e7-aed6-4e727d3173be" (UID: "539e68d2-4ece-47e7-aed6-4e727d3173be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:50.423409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.423376 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq" pod "539e68d2-4ece-47e7-aed6-4e727d3173be" (UID: "539e68d2-4ece-47e7-aed6-4e727d3173be"). InnerVolumeSpecName "kube-api-access-ncfxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:50.432725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.432694 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca" pod "539e68d2-4ece-47e7-aed6-4e727d3173be" (UID: "539e68d2-4ece-47e7-aed6-4e727d3173be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:50.452320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.452288 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:50.538071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.537348 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:50.538071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.537385 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncfxq\" (UniqueName: \"kubernetes.io/projected/539e68d2-4ece-47e7-aed6-4e727d3173be-kube-api-access-ncfxq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:50.538071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.537402 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/539e68d2-4ece-47e7-aed6-4e727d3173be-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:50.638234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.638196 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\") pod \"9f9187ee-2869-426d-9aa3-230ff2d45fee\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " Oct 30 06:07:50.638424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.638353 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\" (UniqueName: \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\") pod \"9f9187ee-2869-426d-9aa3-230ff2d45fee\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " Oct 30 06:07:50.638424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.638386 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\") pod \"9f9187ee-2869-426d-9aa3-230ff2d45fee\" (UID: \"9f9187ee-2869-426d-9aa3-230ff2d45fee\") " Oct 30 06:07:50.638651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.638625 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca" pod "9f9187ee-2869-426d-9aa3-230ff2d45fee" (UID: "9f9187ee-2869-426d-9aa3-230ff2d45fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:50.639214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.639188 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:50.641226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.641191 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca" pod "9f9187ee-2869-426d-9aa3-230ff2d45fee" (UID: "9f9187ee-2869-426d-9aa3-230ff2d45fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:50.641364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.641313 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm" pod "9f9187ee-2869-426d-9aa3-230ff2d45fee" (UID: "9f9187ee-2869-426d-9aa3-230ff2d45fee"). InnerVolumeSpecName "kube-api-access-kskgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:50.740972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.740932 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kskgm\" (UniqueName: \"kubernetes.io/projected/9f9187ee-2869-426d-9aa3-230ff2d45fee-kube-api-access-kskgm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:50.740972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:50.740976 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9187ee-2869-426d-9aa3-230ff2d45fee-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:07:51.212905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.212873 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" Oct 30 06:07:51.212905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.212881 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn" event={"ID":"9f9187ee-2869-426d-9aa3-230ff2d45fee","Type":"ContainerDied","Data":"be81173031b3c9cb2b50fc861bc6c8b192ffe75f05d8b1539948efc327a0e665"} Oct 30 06:07:51.213151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.212935 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="be81173031b3c9cb2b50fc861bc6c8b192ffe75f05d8b1539948efc327a0e665" Oct 30 06:07:51.214909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.214884 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" event={"ID":"539e68d2-4ece-47e7-aed6-4e727d3173be","Type":"ContainerDied","Data":"e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef"} Oct 30 06:07:51.215033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.214910 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86" Oct 30 06:07:51.215033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:51.214921 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e64f1a65c43aae010f8ab93c7fc12e67be4cbd9cf336b7deab68ad4e335e28ef" Oct 30 06:07:51.224834 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:51.224808 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9187ee_2869_426d_9aa3_230ff2d45fee.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:07:56.385639 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:07:56.385603 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:07:59.874931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:07:59.874882 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" containerID="cri-o://359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8" gracePeriod=45 Oct 30 06:08:00.218893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.218860 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:08:00.296689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.296644 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.296913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.296883 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\" (UniqueName: \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.297012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.296989 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.297125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.297107 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.297192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.297168 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.297299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.297283 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\") pod \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\" (UID: \"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af\") " Oct 30 06:08:00.297928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.297892 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:00.298199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.298172 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:00.298525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.298499 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.298615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.298535 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.300344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.300290 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:00.301476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.301441 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:00.303852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.303807 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "kube-api-access-pf5h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:00.310700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.310654 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs" pod "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" (UID: "9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:00.402002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.401950 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.402002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.402010 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.402306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.402028 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pf5h9\" (UniqueName: \"kubernetes.io/projected/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-kube-api-access-pf5h9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.402306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.402057 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:00.420973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.420929 6171 generic.go:356] "Generic (PLEG): container finished" podID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerID="359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8" exitCode=0 Oct 30 06:08:00.421196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.421026 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" Oct 30 06:08:00.421372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.421346 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" event={"ID":"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af","Type":"ContainerDied","Data":"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8"} Oct 30 06:08:00.421440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.421407 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" event={"ID":"9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af","Type":"ContainerDied","Data":"856d881de65d79e77526f23899c84319329c6dce1966421cb2e44fa2cce36f89"} Oct 30 06:08:00.421440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.421433 6171 scope.go:122] "RemoveContainer" containerID="359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8" Oct 30 06:08:00.450420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.449289 6171 scope.go:122] "RemoveContainer" containerID="359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8" Oct 30 06:08:00.450420 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:00.449640 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8\": container with ID starting with 359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8 not found: ID does not exist" containerID="359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8" Oct 30 06:08:00.450420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.449681 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8"} err="failed to get container status \"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8\": rpc error: code = NotFound desc = could not find container \"359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8\": container with ID starting with 359c2cb18bdb60c1dfe53d5aa12ba28140492a87543d0211b1665f6aeecd23b8 not found: ID does not exist" Oct 30 06:08:00.486130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.486076 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:08:00.494588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:00.494558 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc"] Oct 30 06:08:01.157283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.157173 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-aee070/test-model-aee070-kserve-6d67558475-psqdc" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" probeResult="failure" output="Get \"https://10.133.1.4:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:08:01.433443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.433351 6171 generic.go:356] "Generic (PLEG): container finished" podID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerID="3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56" exitCode=0 Oct 30 06:08:01.433443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.433428 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerDied","Data":"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56"} Oct 30 06:08:01.433942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.433923 6171 scope.go:122] "RemoveContainer" containerID="3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56" Oct 30 06:08:01.435930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.435899 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:01.436014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.435975 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:01.436014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.436007 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:08:01.677560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:01.677525 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" path="/var/lib/kubelet/pods/9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af/volumes" Oct 30 06:08:02.004377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:02.004349 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:08:02.448011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:02.447921 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerStarted","Data":"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b"} Oct 30 06:08:02.707918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:02.707761 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:08:02.709163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:02.709129 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:02.709301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:02.709182 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:03.707387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:03.707348 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:03.707907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:03.707427 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:03.865245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:03.865204 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:08:04.652148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:04.644687 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" podUID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" containerName="maas-api" containerID="cri-o://6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3" gracePeriod=30 Oct 30 06:08:04.723137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:04.711035 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:04.723137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:04.711123 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:04.744752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:04.744721 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq"] Oct 30 06:08:04.752936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:04.752911 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f4e35c-lskhq"] Oct 30 06:08:05.165352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.165322 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:08:05.231245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.231162 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\") pod \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " Oct 30 06:08:05.231245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.231217 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\" (UniqueName: \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\") pod \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " Oct 30 06:08:05.231430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.231261 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\") pod \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " Oct 30 06:08:05.231474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.231431 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\") pod \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\" (UID: \"091ceb72-b4b7-4d26-a987-4a12ccb6710d\") " Oct 30 06:08:05.231805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.231781 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca" pod "091ceb72-b4b7-4d26-a987-4a12ccb6710d" (UID: "091ceb72-b4b7-4d26-a987-4a12ccb6710d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:05.232992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.232942 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/091ceb72-b4b7-4d26-a987-4a12ccb6710d-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:05.234203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.234178 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls" pod "091ceb72-b4b7-4d26-a987-4a12ccb6710d" (UID: "091ceb72-b4b7-4d26-a987-4a12ccb6710d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:05.234310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.234228 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz" pod "091ceb72-b4b7-4d26-a987-4a12ccb6710d" (UID: "091ceb72-b4b7-4d26-a987-4a12ccb6710d"). InnerVolumeSpecName "kube-api-access-jkdzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:05.234373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.234352 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs" pod "091ceb72-b4b7-4d26-a987-4a12ccb6710d" (UID: "091ceb72-b4b7-4d26-a987-4a12ccb6710d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:05.347280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.347193 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:05.347280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.347226 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkdzz\" (UniqueName: \"kubernetes.io/projected/091ceb72-b4b7-4d26-a987-4a12ccb6710d-kube-api-access-jkdzz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:05.347280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.347239 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/091ceb72-b4b7-4d26-a987-4a12ccb6710d-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:05.642483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.642393 6171 generic.go:356] "Generic (PLEG): container finished" podID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" containerID="6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3" exitCode=0 Oct 30 06:08:05.642483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.642440 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" Oct 30 06:08:05.642483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.642472 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" event={"ID":"091ceb72-b4b7-4d26-a987-4a12ccb6710d","Type":"ContainerDied","Data":"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3"} Oct 30 06:08:05.642761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.642528 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv" event={"ID":"091ceb72-b4b7-4d26-a987-4a12ccb6710d","Type":"ContainerDied","Data":"12c3af50a7e5ba67c66a8a20eefa764e682ed61ed2adfb28d1f01a919b528851"} Oct 30 06:08:05.642761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.642555 6171 scope.go:122] "RemoveContainer" containerID="6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3" Oct 30 06:08:05.657186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.656960 6171 scope.go:122] "RemoveContainer" containerID="6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3" Oct 30 06:08:05.657303 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:05.657283 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3\": container with ID starting with 6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3 not found: ID does not exist" containerID="6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3" Oct 30 06:08:05.657355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.657311 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3"} err="failed to get container status \"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3\": rpc error: code = NotFound desc = could not find container \"6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3\": container with ID starting with 6eab2b7472e326d943bd8e0511488be2257532dec3b7002ca07d7137b5f965a3 not found: ID does not exist" Oct 30 06:08:05.674823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.674799 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="970d6ecb-00a4-48ae-868a-12f6ddee9820" path="/var/lib/kubelet/pods/970d6ecb-00a4-48ae-868a-12f6ddee9820/volumes" Oct 30 06:08:05.710375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.708697 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:05.710375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.708747 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:05.715778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.715724 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:08:05.724723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:05.724700 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-f4e35c-6d9884696c-rcpxv"] Oct 30 06:08:06.721071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:06.716247 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:06.721071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:06.716307 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:07.678957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:07.678924 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" path="/var/lib/kubelet/pods/091ceb72-b4b7-4d26-a987-4a12ccb6710d/volumes" Oct 30 06:08:07.707646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:07.707597 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:07.707837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:07.707685 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:08.708093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:08.708059 6171 patch_prober.go:29] interesting pod/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" start-of-body= Oct 30 06:08:08.708515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:08.708103 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.17:15021/healthz/ready\": dial tcp 10.133.1.17:15021: connect: connection refused" Oct 30 06:08:09.713108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:09.713034 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:08:09.713585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:09.713565 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:08:09.714328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:09.714304 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:08:12.433590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.433425 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:08:12.681068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.679166 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s"] Oct 30 06:08:12.681068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681065 6171 cpu_manager.go:414] "containerMap: removing container" podUID="539e68d2-4ece-47e7-aed6-4e727d3173be" containerName="revoke-keys" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681089 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="539e68d2-4ece-47e7-aed6-4e727d3173be" containerName="revoke-keys" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681135 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681144 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681179 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9f9187ee-2869-426d-9aa3-230ff2d45fee" containerName="revoke-keys" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681187 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9187ee-2869-426d-9aa3-230ff2d45fee" containerName="revoke-keys" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681211 6171 cpu_manager.go:414] "containerMap: removing container" podUID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" containerName="maas-api" Oct 30 06:08:12.681328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681218 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" containerName="maas-api" Oct 30 06:08:12.681708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681453 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="539e68d2-4ece-47e7-aed6-4e727d3173be" containerName="revoke-keys" Oct 30 06:08:12.681708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681472 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b8c1f7b-4e98-4fa9-96f6-69e6e4b038af" containerName="main" Oct 30 06:08:12.681708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681487 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9187ee-2869-426d-9aa3-230ff2d45fee" containerName="revoke-keys" Oct 30 06:08:12.681708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.681498 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="091ceb72-b4b7-4d26-a987-4a12ccb6710d" containerName="maas-api" Oct 30 06:08:12.733126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.711681 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.756241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.756207 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:12.756716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.756695 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:08:12.756905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.756737 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.757345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.757325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.757629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.757597 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4pxl\" (UniqueName: \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.788005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.787944 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s"] Oct 30 06:08:12.859645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.859613 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.859815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.859735 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4pxl\" (UniqueName: \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.859971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.859947 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.860591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.860570 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.862259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.862238 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:12.887537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:12.887517 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4pxl\" (UniqueName: \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\") pod \"maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:13.117760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:13.117685 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:13.476837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:13.476805 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s"] Oct 30 06:08:13.492711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:13.489806 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 06:08:13.492711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:13.490006 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" containerID="cri-o://9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401" gracePeriod=30 Oct 30 06:08:13.938455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:13.938410 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" event={"ID":"fd43f016-6de1-4db2-a397-a8ff03031391","Type":"ContainerStarted","Data":"7ac2c1ae9de708ce51aacc516717f69844e1a20cfe6982b319cc9d62a41a079e"} Oct 30 06:08:14.023906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.023785 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx"] Oct 30 06:08:14.036399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.035312 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.047422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.047384 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx"] Oct 30 06:08:14.136490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.132649 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.136490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.132835 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.136490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.132887 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k92gs\" (UniqueName: \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.234973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.234937 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k92gs\" (UniqueName: \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.235199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.235188 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.235318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.235303 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.235825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.235801 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.237864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.237842 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.260950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.260915 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k92gs\" (UniqueName: \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\") pod \"maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.440825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.440757 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:14.756430 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:08:14.756401 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e983376_0eb6_4bfb_8fa7_d6e4a46324a3.slice/crio-5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6 WatchSource:0}: Error finding container 5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6: Status 404 returned error can't find the container with id 5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6 Oct 30 06:08:14.758992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.758963 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx"] Oct 30 06:08:14.950458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.950422 6171 generic.go:356] "Generic (PLEG): container finished" podID="fd43f016-6de1-4db2-a397-a8ff03031391" containerID="665010cdb584aa4ba2402e681ffeb0cf0589050520fc6f58905e0d5b82f91fb6" exitCode=0 Oct 30 06:08:14.950650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.950501 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" event={"ID":"fd43f016-6de1-4db2-a397-a8ff03031391","Type":"ContainerDied","Data":"665010cdb584aa4ba2402e681ffeb0cf0589050520fc6f58905e0d5b82f91fb6"} Oct 30 06:08:14.952095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:14.952070 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" event={"ID":"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3","Type":"ContainerStarted","Data":"5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6"} Oct 30 06:08:15.851204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:15.850485 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:08:16.094129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.084357 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" event={"ID":"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3","Type":"ContainerStarted","Data":"e12c394ad653da4e6b03510f135eb6fe2367973149e321c26d67a83bcef4bddf"} Oct 30 06:08:16.171281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.171244 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r"] Oct 30 06:08:16.178682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.178655 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.225311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.225277 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r"] Oct 30 06:08:16.246959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.240467 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" podStartSLOduration=3.240448021 podStartE2EDuration="3.240448021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:16.136430562 +0000 UTC m=+2154.576807756" watchObservedRunningTime="2026-10-30 06:08:16.240448021 +0000 UTC m=+2154.680825217" Oct 30 06:08:16.353091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.352646 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.353091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.352712 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6s9x\" (UniqueName: \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.353091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.352808 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.434680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.434654 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:16.455090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.455056 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.455210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.455104 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6s9x\" (UniqueName: \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.455210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.455158 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.455750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.455729 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.457677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.457656 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.481993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.481967 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6s9x\" (UniqueName: \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\") pod \"maas-api-revoke-keys-e2e-inf-a-aee070-48g9r\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.552951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.552920 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:16.556553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.556534 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\" (UniqueName: \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\") pod \"fd43f016-6de1-4db2-a397-a8ff03031391\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " Oct 30 06:08:16.556760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.556741 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\") pod \"fd43f016-6de1-4db2-a397-a8ff03031391\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " Oct 30 06:08:16.556858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.556791 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\") pod \"fd43f016-6de1-4db2-a397-a8ff03031391\" (UID: \"fd43f016-6de1-4db2-a397-a8ff03031391\") " Oct 30 06:08:16.557159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.557129 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca" pod "fd43f016-6de1-4db2-a397-a8ff03031391" (UID: "fd43f016-6de1-4db2-a397-a8ff03031391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:16.557886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.557819 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:16.559238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.559206 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl" pod "fd43f016-6de1-4db2-a397-a8ff03031391" (UID: "fd43f016-6de1-4db2-a397-a8ff03031391"). InnerVolumeSpecName "kube-api-access-b4pxl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:16.559597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.559572 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca" pod "fd43f016-6de1-4db2-a397-a8ff03031391" (UID: "fd43f016-6de1-4db2-a397-a8ff03031391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:16.660005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.659934 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd43f016-6de1-4db2-a397-a8ff03031391-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:16.660005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.659976 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4pxl\" (UniqueName: \"kubernetes.io/projected/fd43f016-6de1-4db2-a397-a8ff03031391-kube-api-access-b4pxl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:16.872140 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:08:16.872103 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod726eaea3_4d9c_424e_bf8f_48c0d5469ea0.slice/crio-982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b WatchSource:0}: Error finding container 982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b: Status 404 returned error can't find the container with id 982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b Oct 30 06:08:16.873376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:16.873351 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r"] Oct 30 06:08:17.095437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.095404 6171 generic.go:356] "Generic (PLEG): container finished" podID="7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" containerID="e12c394ad653da4e6b03510f135eb6fe2367973149e321c26d67a83bcef4bddf" exitCode=0 Oct 30 06:08:17.095589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.095497 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" event={"ID":"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3","Type":"ContainerDied","Data":"e12c394ad653da4e6b03510f135eb6fe2367973149e321c26d67a83bcef4bddf"} Oct 30 06:08:17.097039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.097016 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" event={"ID":"726eaea3-4d9c-424e-bf8f-48c0d5469ea0","Type":"ContainerStarted","Data":"982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b"} Oct 30 06:08:17.098513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.098490 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" event={"ID":"fd43f016-6de1-4db2-a397-a8ff03031391","Type":"ContainerDied","Data":"7ac2c1ae9de708ce51aacc516717f69844e1a20cfe6982b319cc9d62a41a079e"} Oct 30 06:08:17.098610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.098533 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac2c1ae9de708ce51aacc516717f69844e1a20cfe6982b319cc9d62a41a079e" Oct 30 06:08:17.098610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:17.098498 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s" Oct 30 06:08:18.125754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.125709 6171 generic.go:356] "Generic (PLEG): container finished" podID="726eaea3-4d9c-424e-bf8f-48c0d5469ea0" containerID="c54d94efe203e32745c98227fa1cbdc72e9414675cae75205a4a13debd12337a" exitCode=0 Oct 30 06:08:18.126262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.125785 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" event={"ID":"726eaea3-4d9c-424e-bf8f-48c0d5469ea0","Type":"ContainerDied","Data":"c54d94efe203e32745c98227fa1cbdc72e9414675cae75205a4a13debd12337a"} Oct 30 06:08:18.407066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.404139 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:18.468272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.468230 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\" (UniqueName: \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\") pod \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " Oct 30 06:08:18.468491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.468293 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\") pod \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " Oct 30 06:08:18.468710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.468691 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\") pod \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\" (UID: \"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3\") " Oct 30 06:08:18.469191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.469107 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca" pod "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" (UID: "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:18.469501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.469481 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.475796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.475753 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca" pod "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" (UID: "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:18.476139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.476112 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs" pod "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" (UID: "7e983376-0eb6-4bfb-8fa7-d6e4a46324a3"). InnerVolumeSpecName "kube-api-access-k92gs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:18.571073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.570989 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k92gs\" (UniqueName: \"kubernetes.io/projected/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-kube-api-access-k92gs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.571073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.571020 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.784063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.784022 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:08:18.877106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877010 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877227 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877298 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877363 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877420 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877529 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877537 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:18.877654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877627 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.877715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877683 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.878074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877777 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:18.878074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.877794 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:18.878074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.878019 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:18.878322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.878128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:18.878322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.878197 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\") pod \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\" (UID: \"dfde1455-eedf-4b3c-bca0-e8a9ea76df05\") " Oct 30 06:08:18.879032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.879013 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.879235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.879035 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.879235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.879074 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.879235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.879086 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.879235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.879097 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.880617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.880593 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "kube-api-access-dtx79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:18.883529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.883479 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:18.883621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.883528 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:18.883621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.883553 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo" pod "dfde1455-eedf-4b3c-bca0-e8a9ea76df05" (UID: "dfde1455-eedf-4b3c-bca0-e8a9ea76df05"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:18.980395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.980357 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtx79\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-kube-api-access-dtx79\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.980395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.980387 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.980395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.980400 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:18.980604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:18.980411 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dfde1455-eedf-4b3c-bca0-e8a9ea76df05-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:19.139387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.139309 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" Oct 30 06:08:19.139387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.139343 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx" event={"ID":"7e983376-0eb6-4bfb-8fa7-d6e4a46324a3","Type":"ContainerDied","Data":"5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6"} Oct 30 06:08:19.139871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.139406 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ece6b598aa296961aa5185966b0f96d8e36d08fba25f6587f154467784aa5d6" Oct 30 06:08:19.141571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.141545 6171 generic.go:356] "Generic (PLEG): container finished" podID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerID="9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401" exitCode=0 Oct 30 06:08:19.141700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.141590 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" Oct 30 06:08:19.141700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.141636 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerDied","Data":"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401"} Oct 30 06:08:19.141700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.141690 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs" event={"ID":"dfde1455-eedf-4b3c-bca0-e8a9ea76df05","Type":"ContainerDied","Data":"d1200f9427b869fe4fe568dbb3db69696380a8020a8374f618ec6a191e7e085b"} Oct 30 06:08:19.141842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.141725 6171 scope.go:122] "RemoveContainer" containerID="9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401" Oct 30 06:08:19.158659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.158639 6171 scope.go:122] "RemoveContainer" containerID="26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c" Oct 30 06:08:19.173588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.173561 6171 scope.go:122] "RemoveContainer" containerID="9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401" Oct 30 06:08:19.173878 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:19.173860 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401\": container with ID starting with 9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401 not found: ID does not exist" containerID="9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401" Oct 30 06:08:19.173954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.173887 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401"} err="failed to get container status \"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401\": rpc error: code = NotFound desc = could not find container \"9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401\": container with ID starting with 9cf7af304a9ae150904700e0d8a2e85ec34580e30d811835118d5b079b5a3401 not found: ID does not exist" Oct 30 06:08:19.173954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.173904 6171 scope.go:122] "RemoveContainer" containerID="26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c" Oct 30 06:08:19.174261 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:19.174232 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c\": container with ID starting with 26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c not found: ID does not exist" containerID="26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c" Oct 30 06:08:19.174384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.174266 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c"} err="failed to get container status \"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c\": rpc error: code = NotFound desc = could not find container \"26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c\": container with ID starting with 26473bff8995d24904257e07f295377ff5da24de3c0ced0abab7a98c64ab337c not found: ID does not exist" Oct 30 06:08:19.231147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.228422 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 06:08:19.241496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.241466 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-f4e35c-openshift-default-5b758f58b7-wshzs"] Oct 30 06:08:19.350596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.350573 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:19.498065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.498015 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\" (UniqueName: \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\") pod \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " Oct 30 06:08:19.498282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.498172 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\") pod \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " Oct 30 06:08:19.498359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.498286 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\") pod \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\" (UID: \"726eaea3-4d9c-424e-bf8f-48c0d5469ea0\") " Oct 30 06:08:19.523822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.515249 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca" pod "726eaea3-4d9c-424e-bf8f-48c0d5469ea0" (UID: "726eaea3-4d9c-424e-bf8f-48c0d5469ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:19.543099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.532382 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x" pod "726eaea3-4d9c-424e-bf8f-48c0d5469ea0" (UID: "726eaea3-4d9c-424e-bf8f-48c0d5469ea0"). InnerVolumeSpecName "kube-api-access-b6s9x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:19.543099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.540171 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca" pod "726eaea3-4d9c-424e-bf8f-48c0d5469ea0" (UID: "726eaea3-4d9c-424e-bf8f-48c0d5469ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:19.604115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.603580 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:19.604115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.603609 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6s9x\" (UniqueName: \"kubernetes.io/projected/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-kube-api-access-b6s9x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:19.604115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.603618 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/726eaea3-4d9c-424e-bf8f-48c0d5469ea0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:19.676592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:19.676556 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" path="/var/lib/kubelet/pods/dfde1455-eedf-4b3c-bca0-e8a9ea76df05/volumes" Oct 30 06:08:20.153030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:20.153000 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" Oct 30 06:08:20.153570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:20.153003 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r" event={"ID":"726eaea3-4d9c-424e-bf8f-48c0d5469ea0","Type":"ContainerDied","Data":"982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b"} Oct 30 06:08:20.153570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:20.153149 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="982fd5f978468cd97117b3a430ba781bb6ace9b24057baa3ec99f3dad27c5f1b" Oct 30 06:08:23.148022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:23.147985 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:08:23.174897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:23.174864 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:08:24.467464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:24.467426 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:08:26.695078 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:26.686586 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:08:26.819006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:26.818970 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s"] Oct 30 06:08:26.834115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:26.834088 6171 scope.go:122] "RemoveContainer" containerID="5d825ffd07018c3cec84b9b88ec06854a8ba5ea2c94f7abf52747d71471cbb25" Oct 30 06:08:26.893479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:26.891168 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-387b16-6pd8s"] Oct 30 06:08:26.964784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:26.964751 6171 scope.go:122] "RemoveContainer" containerID="9a74ddafa10cf2abafaba8c8533718184d5bf4253f3bf8d3466d6b90a15d4864" Oct 30 06:08:27.055417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:27.055389 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:08:27.087597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:27.087564 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:08:27.457866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:27.457829 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" containerID="cri-o://1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1" gracePeriod=45 Oct 30 06:08:27.681992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:27.681958 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd43f016-6de1-4db2-a397-a8ff03031391" path="/var/lib/kubelet/pods/fd43f016-6de1-4db2-a397-a8ff03031391/volumes" Oct 30 06:08:28.108775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.108747 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:08:28.148449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.148388 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" podUID="1908ce98-e066-4e9a-a314-119aebe9b20c" containerName="payload-processing" containerID="cri-o://525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60" gracePeriod=25 Oct 30 06:08:28.176016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.175960 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" containerID="cri-o://252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370" gracePeriod=25 Oct 30 06:08:28.211303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211266 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.211489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211327 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\" (UniqueName: \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.211489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211415 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.211628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211554 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\" (UniqueName: \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.211702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211678 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:28.211775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211761 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.211822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211789 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\") pod \"202a3a3b-fb27-4a44-85b7-ccac61e07326\" (UID: \"202a3a3b-fb27-4a44-85b7-ccac61e07326\") " Oct 30 06:08:28.212027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.211992 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:28.212162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.212139 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:28.212680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.212662 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.212739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.212682 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.212739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.212690 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.213775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.213756 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:28.214234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.214214 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:28.214301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.214284 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg" pod "202a3a3b-fb27-4a44-85b7-ccac61e07326" (UID: "202a3a3b-fb27-4a44-85b7-ccac61e07326"). InnerVolumeSpecName "kube-api-access-8jkdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:28.314522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.314432 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/202a3a3b-fb27-4a44-85b7-ccac61e07326-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.314522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.314466 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/202a3a3b-fb27-4a44-85b7-ccac61e07326-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.314522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.314478 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkdg\" (UniqueName: \"kubernetes.io/projected/202a3a3b-fb27-4a44-85b7-ccac61e07326-kube-api-access-8jkdg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.419156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.419129 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:08:28.538074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.530904 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\") pod \"1908ce98-e066-4e9a-a314-119aebe9b20c\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " Oct 30 06:08:28.538074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.531013 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\" (UniqueName: \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\") pod \"1908ce98-e066-4e9a-a314-119aebe9b20c\" (UID: \"1908ce98-e066-4e9a-a314-119aebe9b20c\") " Oct 30 06:08:28.594490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.594360 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume" pod "1908ce98-e066-4e9a-a314-119aebe9b20c" (UID: "1908ce98-e066-4e9a-a314-119aebe9b20c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:28.620290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.620257 6171 generic.go:356] "Generic (PLEG): container finished" podID="1908ce98-e066-4e9a-a314-119aebe9b20c" containerID="525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60" exitCode=0 Oct 30 06:08:28.620830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.620810 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" Oct 30 06:08:28.639070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.630282 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" event={"ID":"1908ce98-e066-4e9a-a314-119aebe9b20c","Type":"ContainerDied","Data":"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60"} Oct 30 06:08:28.639070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.630359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp" event={"ID":"1908ce98-e066-4e9a-a314-119aebe9b20c","Type":"ContainerDied","Data":"c7af8a8105aab7c5ba398a0b15c18bd22b52ce8dbcc416e2aaeab67bf1f048fb"} Oct 30 06:08:28.639070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.630392 6171 scope.go:122] "RemoveContainer" containerID="525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60" Oct 30 06:08:28.648007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.647978 6171 generic.go:356] "Generic (PLEG): container finished" podID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerID="1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1" exitCode=0 Oct 30 06:08:28.648206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.648105 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" event={"ID":"202a3a3b-fb27-4a44-85b7-ccac61e07326","Type":"ContainerDied","Data":"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1"} Oct 30 06:08:28.648206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.648139 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" Oct 30 06:08:28.648312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.648155 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp" event={"ID":"202a3a3b-fb27-4a44-85b7-ccac61e07326","Type":"ContainerDied","Data":"4f75e19fe3178f03f0227093eab06cc4d95dc887089b548c3a66889ff2317e68"} Oct 30 06:08:28.649164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.648907 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp" pod "1908ce98-e066-4e9a-a314-119aebe9b20c" (UID: "1908ce98-e066-4e9a-a314-119aebe9b20c"). InnerVolumeSpecName "kube-api-access-gtmrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:28.664695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.664672 6171 scope.go:122] "RemoveContainer" containerID="525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60" Oct 30 06:08:28.665125 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:28.665024 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60\": container with ID starting with 525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60 not found: ID does not exist" containerID="525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60" Oct 30 06:08:28.665220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.665160 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60"} err="failed to get container status \"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60\": rpc error: code = NotFound desc = could not find container \"525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60\": container with ID starting with 525b595f7c19d4bf5de1cbfbf794cf721ceb22197f12b5d9fc1ad00fc72dad60 not found: ID does not exist" Oct 30 06:08:28.665220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.665186 6171 scope.go:122] "RemoveContainer" containerID="1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1" Oct 30 06:08:28.678241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.678220 6171 scope.go:122] "RemoveContainer" containerID="1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1" Oct 30 06:08:28.678572 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:28.678547 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1\": container with ID starting with 1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1 not found: ID does not exist" containerID="1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1" Oct 30 06:08:28.678652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.678580 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1"} err="failed to get container status \"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1\": rpc error: code = NotFound desc = could not find container \"1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1\": container with ID starting with 1de4dfa1ab7e92d3940b45bb8a7e827d81d20622948b74efdc0acde6e23b15f1 not found: ID does not exist" Oct 30 06:08:28.684453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.684419 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1908ce98-e066-4e9a-a314-119aebe9b20c-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.684453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.684447 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtmrp\" (UniqueName: \"kubernetes.io/projected/1908ce98-e066-4e9a-a314-119aebe9b20c-kube-api-access-gtmrp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:28.730326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.730285 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:08:28.730609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.730594 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-7ad9ad/test-model-7ad9ad-kserve-749f974487-lqgcp"] Oct 30 06:08:28.992016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.991977 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:08:28.997310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:28.997283 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-387b16-569d556879-gmrbp"] Oct 30 06:08:29.468120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.468033 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" podUID="62aee73c-f5fa-4579-91ff-b1f69df57228" containerName="payload-processing" containerID="cri-o://7d6d776a6a1cc63c5ed86ed935d2bdbcdc9050cb5b6d3328ae309d88a78d4e2b" gracePeriod=25 Oct 30 06:08:29.672953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.672435 6171 generic.go:356] "Generic (PLEG): container finished" podID="62aee73c-f5fa-4579-91ff-b1f69df57228" containerID="7d6d776a6a1cc63c5ed86ed935d2bdbcdc9050cb5b6d3328ae309d88a78d4e2b" exitCode=0 Oct 30 06:08:29.680295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.680110 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1908ce98-e066-4e9a-a314-119aebe9b20c" path="/var/lib/kubelet/pods/1908ce98-e066-4e9a-a314-119aebe9b20c/volumes" Oct 30 06:08:29.684093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.680811 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" path="/var/lib/kubelet/pods/202a3a3b-fb27-4a44-85b7-ccac61e07326/volumes" Oct 30 06:08:29.684093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.681320 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" event={"ID":"62aee73c-f5fa-4579-91ff-b1f69df57228","Type":"ContainerDied","Data":"7d6d776a6a1cc63c5ed86ed935d2bdbcdc9050cb5b6d3328ae309d88a78d4e2b"} Oct 30 06:08:29.770609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.770577 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:08:29.806587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.806550 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\") pod \"62aee73c-f5fa-4579-91ff-b1f69df57228\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " Oct 30 06:08:29.806762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.806686 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\" (UniqueName: \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\") pod \"62aee73c-f5fa-4579-91ff-b1f69df57228\" (UID: \"62aee73c-f5fa-4579-91ff-b1f69df57228\") " Oct 30 06:08:29.806955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.806926 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume" pod "62aee73c-f5fa-4579-91ff-b1f69df57228" (UID: "62aee73c-f5fa-4579-91ff-b1f69df57228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:29.807486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.807463 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/62aee73c-f5fa-4579-91ff-b1f69df57228-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:29.808974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.808949 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx" pod "62aee73c-f5fa-4579-91ff-b1f69df57228" (UID: "62aee73c-f5fa-4579-91ff-b1f69df57228"). InnerVolumeSpecName "kube-api-access-mqxbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:29.909304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:29.909266 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxbx\" (UniqueName: \"kubernetes.io/projected/62aee73c-f5fa-4579-91ff-b1f69df57228-kube-api-access-mqxbx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:30.689081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.689021 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" Oct 30 06:08:30.689081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.689066 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd" event={"ID":"62aee73c-f5fa-4579-91ff-b1f69df57228","Type":"ContainerDied","Data":"7c041cafc011b55b03ea5c86c83bba8205296d47bdff3b79e05cd36da2c83552"} Oct 30 06:08:30.689539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.689140 6171 scope.go:122] "RemoveContainer" containerID="7d6d776a6a1cc63c5ed86ed935d2bdbcdc9050cb5b6d3328ae309d88a78d4e2b" Oct 30 06:08:30.750422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.750383 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:08:30.764874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.764835 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-3cc570-7cc44d69c-9dqdd"] Oct 30 06:08:30.885554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:30.885504 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" containerID="cri-o://e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b" gracePeriod=45 Oct 30 06:08:31.224908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.224872 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:08:31.349666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349576 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\" (UniqueName: \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.349859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349694 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.349859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349746 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.349859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349798 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.350068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349916 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.350068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.349991 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:31.350068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.350027 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:31.350249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.350036 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\") pod \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\" (UID: \"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd\") " Oct 30 06:08:31.350249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.350225 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:31.351014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.350989 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.351014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.351016 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.351238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.351025 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.352328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.352300 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:31.352449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.352371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "kube-api-access-xjzch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:31.352637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.352613 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs" pod "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" (UID: "6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:31.453086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.452984 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjzch\" (UniqueName: \"kubernetes.io/projected/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-kube-api-access-xjzch\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.453086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.453022 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.453086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.453032 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:31.677902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.677815 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62aee73c-f5fa-4579-91ff-b1f69df57228" path="/var/lib/kubelet/pods/62aee73c-f5fa-4579-91ff-b1f69df57228/volumes" Oct 30 06:08:31.702034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.701994 6171 generic.go:356] "Generic (PLEG): container finished" podID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerID="e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b" exitCode=0 Oct 30 06:08:31.702593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.702062 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" event={"ID":"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd","Type":"ContainerDied","Data":"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b"} Oct 30 06:08:31.702593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.702085 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" Oct 30 06:08:31.702593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.702115 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d" event={"ID":"6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd","Type":"ContainerDied","Data":"c09ff7f69df1deecb929327aaebd0f66e5f2e686d1840975800fd015b1f6ce27"} Oct 30 06:08:31.702593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.702137 6171 scope.go:122] "RemoveContainer" containerID="e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b" Oct 30 06:08:31.722834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.722809 6171 scope.go:122] "RemoveContainer" containerID="e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b" Oct 30 06:08:31.723163 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:31.723140 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b\": container with ID starting with e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b not found: ID does not exist" containerID="e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b" Oct 30 06:08:31.723256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.723170 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b"} err="failed to get container status \"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b\": rpc error: code = NotFound desc = could not find container \"e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b\": container with ID starting with e0e95ac071ab9868cdd71f7c32317355882ea7fa90aecdec1fa95d9164d2628b not found: ID does not exist" Oct 30 06:08:31.764147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.764105 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:08:31.770110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:31.770073 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-c54211/test-model-c54211-kserve-5b6b668ff-jd95d"] Oct 30 06:08:32.055953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:32.055893 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" containerID="cri-o://a1f2adebf289318c93646b2499edc347fb74da7e7b2fc6e41b819ce2131f4bfb" gracePeriod=25 Oct 30 06:08:32.088861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:32.088723 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" containerID="cri-o://52d8e5ed7a2321808e7de012dbc6dc112e6f12cb5c0d0add180d824406e8be8a" gracePeriod=25 Oct 30 06:08:33.680984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:33.680917 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" path="/var/lib/kubelet/pods/6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd/volumes" Oct 30 06:08:34.197155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.197103 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:08:34.197466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.197408 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" containerID="cri-o://2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772" gracePeriod=30 Oct 30 06:08:34.214429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.214388 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:08:34.214671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.214594 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" containerID="cri-o://68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede" gracePeriod=30 Oct 30 06:08:34.227385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.227327 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:08:34.227665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.227631 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" containerID="cri-o://a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4" gracePeriod=30 Oct 30 06:08:34.236473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.236423 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:08:34.236767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.236736 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" containerID="cri-o://5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992" gracePeriod=30 Oct 30 06:08:34.761984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.761940 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b"] Oct 30 06:08:34.762979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.762956 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1908ce98-e066-4e9a-a314-119aebe9b20c" containerName="payload-processing" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.762982 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1908ce98-e066-4e9a-a314-119aebe9b20c" containerName="payload-processing" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763001 6171 cpu_manager.go:414] "containerMap: removing container" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763007 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763014 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fd43f016-6de1-4db2-a397-a8ff03031391" containerName="revoke-keys" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763019 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd43f016-6de1-4db2-a397-a8ff03031391" containerName="revoke-keys" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763033 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" containerName="revoke-keys" Oct 30 06:08:34.763068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763038 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" containerName="revoke-keys" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763076 6171 cpu_manager.go:414] "containerMap: removing container" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763083 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763135 6171 cpu_manager.go:414] "containerMap: removing container" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763143 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763155 6171 cpu_manager.go:414] "containerMap: removing container" podUID="726eaea3-4d9c-424e-bf8f-48c0d5469ea0" containerName="revoke-keys" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763159 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="726eaea3-4d9c-424e-bf8f-48c0d5469ea0" containerName="revoke-keys" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763174 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763179 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763196 6171 cpu_manager.go:414] "containerMap: removing container" podUID="62aee73c-f5fa-4579-91ff-b1f69df57228" containerName="payload-processing" Oct 30 06:08:34.763320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763201 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62aee73c-f5fa-4579-91ff-b1f69df57228" containerName="payload-processing" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763365 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1908ce98-e066-4e9a-a314-119aebe9b20c" containerName="payload-processing" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763377 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763382 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd43f016-6de1-4db2-a397-a8ff03031391" containerName="revoke-keys" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763391 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" containerName="revoke-keys" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763399 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="202a3a3b-fb27-4a44-85b7-ccac61e07326" containerName="main" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763409 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="726eaea3-4d9c-424e-bf8f-48c0d5469ea0" containerName="revoke-keys" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763417 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2d5bdc-31c7-4fa7-97c2-3435a4f9f8fd" containerName="main" Oct 30 06:08:34.763641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.763426 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="62aee73c-f5fa-4579-91ff-b1f69df57228" containerName="payload-processing" Oct 30 06:08:34.769353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.769323 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.788180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.787980 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b"] Oct 30 06:08:34.831214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.831160 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.831503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.831476 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6pk\" (UniqueName: \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.831613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.831586 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.856378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.856341 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:34.856573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.856395 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:08:34.874600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.874548 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:34.933359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.933316 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6pk\" (UniqueName: \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.933576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.933407 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.933664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.933643 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.934284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.934263 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.936357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.936322 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:34.959255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:34.959216 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6pk\" (UniqueName: \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\") pod \"maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:35.068861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:35.068774 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:35.069191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:35.068858 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:35.093070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:35.093019 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:35.451254 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:08:35.451213 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dd274c_e696_46e9_8da0_b7ab6e0102b7.slice/crio-2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb WatchSource:0}: Error finding container 2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb: Status 404 returned error can't find the container with id 2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb Oct 30 06:08:35.455312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:35.455278 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b"] Oct 30 06:08:35.762424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:35.762367 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" event={"ID":"10dd274c-e696-46e9-8da0-b7ab6e0102b7","Type":"ContainerStarted","Data":"2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb"} Oct 30 06:08:36.773974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:36.773932 6171 generic.go:356] "Generic (PLEG): container finished" podID="10dd274c-e696-46e9-8da0-b7ab6e0102b7" containerID="9954cb9b3d2ec76bcca5ed6189aab746819a59e5e6c9d14d41cd9ea2b550c9cf" exitCode=0 Oct 30 06:08:36.774470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:36.774017 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" event={"ID":"10dd274c-e696-46e9-8da0-b7ab6e0102b7","Type":"ContainerDied","Data":"9954cb9b3d2ec76bcca5ed6189aab746819a59e5e6c9d14d41cd9ea2b550c9cf"} Oct 30 06:08:37.676602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:37.675964 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:37.676602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:37.676083 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:37.678696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:37.678585 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:37.678696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:37.678633 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:37.988063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:37.988020 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:38.090015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.089982 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\") pod \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " Oct 30 06:08:38.090260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.090035 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\" (UniqueName: \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\") pod \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " Oct 30 06:08:38.090444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.090421 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\") pod \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\" (UID: \"10dd274c-e696-46e9-8da0-b7ab6e0102b7\") " Oct 30 06:08:38.090858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.090826 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca" pod "10dd274c-e696-46e9-8da0-b7ab6e0102b7" (UID: "10dd274c-e696-46e9-8da0-b7ab6e0102b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:38.091650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.091627 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:38.095176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.095130 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca" pod "10dd274c-e696-46e9-8da0-b7ab6e0102b7" (UID: "10dd274c-e696-46e9-8da0-b7ab6e0102b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:38.095553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.095524 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk" pod "10dd274c-e696-46e9-8da0-b7ab6e0102b7" (UID: "10dd274c-e696-46e9-8da0-b7ab6e0102b7"). InnerVolumeSpecName "kube-api-access-fx6pk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:38.193596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.193553 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dd274c-e696-46e9-8da0-b7ab6e0102b7-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:38.193596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.193592 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6pk\" (UniqueName: \"kubernetes.io/projected/10dd274c-e696-46e9-8da0-b7ab6e0102b7-kube-api-access-fx6pk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:38.797320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.797290 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" Oct 30 06:08:38.797320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.797317 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b" event={"ID":"10dd274c-e696-46e9-8da0-b7ab6e0102b7","Type":"ContainerDied","Data":"2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb"} Oct 30 06:08:38.797551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:38.797361 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ff1355fd4636d335079cdd8e294f618577741deb680f74c91746a2bbd3920eb" Oct 30 06:08:39.300161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.299806 6171 patch_prober.go:29] interesting pod/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.243:15021/healthz/ready\": read tcp 10.133.0.2:45844->10.133.0.243:15021: read: connection reset by peer" start-of-body= Oct 30 06:08:39.300161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.299870 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.243:15021/healthz/ready\": read tcp 10.133.0.2:45844->10.133.0.243:15021: read: connection reset by peer" Oct 30 06:08:39.431646 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.430916 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678700b0_4562_45e6_ad2a_f987fac4b574.slice/crio-a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b14f741_7915_4c5f_984c_0fb0436a5959.slice/crio-2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod678700b0_4562_45e6_ad2a_f987fac4b574.slice/crio-conmon-a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4571e3a6_c10d_4b7c_bdb3_2be7276a951f.slice/crio-conmon-68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b14f741_7915_4c5f_984c_0fb0436a5959.slice/crio-conmon-2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:08:39.603817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.602103 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:08:39.730886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.730847 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731081 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731154 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731213 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731379 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731436 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.731504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731466 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731523 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731584 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731716 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\") pod \"3b14f741-7915-4c5f-984c-0fb0436a5959\" (UID: \"3b14f741-7915-4c5f-984c-0fb0436a5959\") " Oct 30 06:08:39.731805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.731752 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:39.733735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.733681 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.734459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.734432 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.734591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.734469 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.734591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.734486 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b14f741-7915-4c5f-984c-0fb0436a5959-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.737026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.736995 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.737026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.737014 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:39.737389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.737361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.737499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.737430 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.738551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.738527 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:08:39.740562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.740538 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.740689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.740628 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg" pod "3b14f741-7915-4c5f-984c-0fb0436a5959" (UID: "3b14f741-7915-4c5f-984c-0fb0436a5959"). InnerVolumeSpecName "kube-api-access-fqfdg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.742699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.742669 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:08:39.752414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.752387 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:08:39.811509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.811405 6171 generic.go:356] "Generic (PLEG): container finished" podID="678700b0-4562-45e6-ad2a-f987fac4b574" containerID="a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4" exitCode=0 Oct 30 06:08:39.811701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.811658 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" Oct 30 06:08:39.811840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.811819 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" event={"ID":"678700b0-4562-45e6-ad2a-f987fac4b574","Type":"ContainerDied","Data":"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4"} Oct 30 06:08:39.811906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.811861 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp" event={"ID":"678700b0-4562-45e6-ad2a-f987fac4b574","Type":"ContainerDied","Data":"e4d0e7308e293088e75cdb1351f5a8827ac680eb7118c478f69900492c69fec5"} Oct 30 06:08:39.811906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.811884 6171 scope.go:122] "RemoveContainer" containerID="a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4" Oct 30 06:08:39.814657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.814631 6171 generic.go:356] "Generic (PLEG): container finished" podID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerID="5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992" exitCode=0 Oct 30 06:08:39.814856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.814839 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" event={"ID":"ac3aa7f6-fbc6-484f-9764-b25f17edb471","Type":"ContainerDied","Data":"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992"} Oct 30 06:08:39.814971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.814891 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" event={"ID":"ac3aa7f6-fbc6-484f-9764-b25f17edb471","Type":"ContainerDied","Data":"8c88efba3b1206e6dc22e91fbe73ff17d4295d894e7329e643dbac51f6b5fc20"} Oct 30 06:08:39.815145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.814984 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q" Oct 30 06:08:39.817467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.817442 6171 generic.go:356] "Generic (PLEG): container finished" podID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerID="2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772" exitCode=0 Oct 30 06:08:39.817658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.817638 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerDied","Data":"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772"} Oct 30 06:08:39.817735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.817689 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" event={"ID":"3b14f741-7915-4c5f-984c-0fb0436a5959","Type":"ContainerDied","Data":"4ef7e09f533a6ee7443edb5fafbe037bf45b31daf911ab711b336ed55e1055f0"} Oct 30 06:08:39.817813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.817798 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr" Oct 30 06:08:39.820586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.820559 6171 generic.go:356] "Generic (PLEG): container finished" podID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerID="68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede" exitCode=0 Oct 30 06:08:39.820700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.820605 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" event={"ID":"4571e3a6-c10d-4b7c-bdb3-2be7276a951f","Type":"ContainerDied","Data":"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede"} Oct 30 06:08:39.820700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.820645 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" event={"ID":"4571e3a6-c10d-4b7c-bdb3-2be7276a951f","Type":"ContainerDied","Data":"10ea511c6da885dbc941f26903db95e8c16f0abff66dc94698e58b5e053c596a"} Oct 30 06:08:39.820842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.820722 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj" Oct 30 06:08:39.828678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.828646 6171 scope.go:122] "RemoveContainer" containerID="a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4" Oct 30 06:08:39.829141 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.829038 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4\": container with ID starting with a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4 not found: ID does not exist" containerID="a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4" Oct 30 06:08:39.829227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.829154 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4"} err="failed to get container status \"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4\": rpc error: code = NotFound desc = could not find container \"a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4\": container with ID starting with a8e6424c1d7b3e12b3c0f2587566a0860b11949bdfd85b4796cac8cea27f3ed4 not found: ID does not exist" Oct 30 06:08:39.829227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.829176 6171 scope.go:122] "RemoveContainer" containerID="5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992" Oct 30 06:08:39.835966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.835939 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.835966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.835979 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.836243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836222 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.836323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836257 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.836398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836345 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.836398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836357 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.836398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836376 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.836574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836409 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.836574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836439 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.836574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.836574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836535 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836577 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836587 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836651 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836723 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836806 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836923 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.836979 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837065 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837127 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837157 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\") pod \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\" (UID: \"ac3aa7f6-fbc6-484f-9764-b25f17edb471\") " Oct 30 06:08:39.837431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837277 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\") pod \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\" (UID: \"4571e3a6-c10d-4b7c-bdb3-2be7276a951f\") " Oct 30 06:08:39.838353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.837748 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:39.838353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.838012 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.838714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.838514 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.839290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.838938 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.839290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.839128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:39.840391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.839649 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840761 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840789 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840804 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840816 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840828 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840839 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840851 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840865 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqfdg\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-kube-api-access-fqfdg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840878 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840891 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840907 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840919 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840931 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840941 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840952 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840965 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b14f741-7915-4c5f-984c-0fb0436a5959-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840883 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "kube-api-access-tgn4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840978 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.840996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.840969 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "kube-api-access-d2wkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.842421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.842380 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:39.843727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.843691 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.844739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.844664 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.846387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.846346 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo" pod "4571e3a6-c10d-4b7c-bdb3-2be7276a951f" (UID: "4571e3a6-c10d-4b7c-bdb3-2be7276a951f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:39.846387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.846373 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token" pod "ac3aa7f6-fbc6-484f-9764-b25f17edb471" (UID: "ac3aa7f6-fbc6-484f-9764-b25f17edb471"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.846879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.846855 6171 scope.go:122] "RemoveContainer" containerID="5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992" Oct 30 06:08:39.847272 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.847246 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992\": container with ID starting with 5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992 not found: ID does not exist" containerID="5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992" Oct 30 06:08:39.847391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.847284 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992"} err="failed to get container status \"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992\": rpc error: code = NotFound desc = could not find container \"5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992\": container with ID starting with 5ac1a26105bf21e7df7fce45f9db117fe546e9f75c85817f32dbb98098ecc992 not found: ID does not exist" Oct 30 06:08:39.847391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.847310 6171 scope.go:122] "RemoveContainer" containerID="2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772" Oct 30 06:08:39.861713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.861683 6171 scope.go:122] "RemoveContainer" containerID="4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1" Oct 30 06:08:39.879744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.879716 6171 scope.go:122] "RemoveContainer" containerID="2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772" Oct 30 06:08:39.880210 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.880180 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772\": container with ID starting with 2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772 not found: ID does not exist" containerID="2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772" Oct 30 06:08:39.880353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.880220 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772"} err="failed to get container status \"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772\": rpc error: code = NotFound desc = could not find container \"2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772\": container with ID starting with 2349b71c3e8510ed400b7b78971dd90fb16e858646b94a3fce45a6995fa28772 not found: ID does not exist" Oct 30 06:08:39.880353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.880239 6171 scope.go:122] "RemoveContainer" containerID="4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1" Oct 30 06:08:39.880589 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.880568 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1\": container with ID starting with 4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1 not found: ID does not exist" containerID="4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1" Oct 30 06:08:39.880676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.880596 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1"} err="failed to get container status \"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1\": rpc error: code = NotFound desc = could not find container \"4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1\": container with ID starting with 4ace312046ca779eab95f867695de4fbd6a7f963740abbc1b7b5b086651642d1 not found: ID does not exist" Oct 30 06:08:39.880676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.880614 6171 scope.go:122] "RemoveContainer" containerID="68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede" Oct 30 06:08:39.894886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.894864 6171 scope.go:122] "RemoveContainer" containerID="68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede" Oct 30 06:08:39.895359 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:39.895321 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede\": container with ID starting with 68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede not found: ID does not exist" containerID="68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede" Oct 30 06:08:39.895510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.895353 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede"} err="failed to get container status \"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede\": rpc error: code = NotFound desc = could not find container \"68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede\": container with ID starting with 68d869b6fe708dc555ab0562e9e018156634ae971546113b26d3b1a84abb4ede not found: ID does not exist" Oct 30 06:08:39.942022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.941985 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942175 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942241 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942310 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942349 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.942496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942377 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942408 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942544 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942677 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942699 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\") pod \"678700b0-4562-45e6-ad2a-f987fac4b574\" (UID: \"678700b0-4562-45e6-ad2a-f987fac4b574\") " Oct 30 06:08:39.942829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.942796 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:39.943167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943140 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.943464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943423 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.943464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943441 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943677 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943704 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2wkd\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-kube-api-access-d2wkd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943722 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943736 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/678700b0-4562-45e6-ad2a-f987fac4b574-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943750 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgn4n\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-kube-api-access-tgn4n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943764 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4571e3a6-c10d-4b7c-bdb3-2be7276a951f-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943778 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943793 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943808 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943820 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943834 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac3aa7f6-fbc6-484f-9764-b25f17edb471-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.943884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.943846 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:39.946365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.946280 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:39.946365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.946325 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:39.946365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.946337 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "kube-api-access-rzxr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.946365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.946329 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token" pod "678700b0-4562-45e6-ad2a-f987fac4b574" (UID: "678700b0-4562-45e6-ad2a-f987fac4b574"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:39.956637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.956602 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:08:39.962254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:39.962221 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-d5nwr"] Oct 30 06:08:40.045668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.045630 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/678700b0-4562-45e6-ad2a-f987fac4b574-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:40.045668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.045664 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:40.045668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.045673 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/678700b0-4562-45e6-ad2a-f987fac4b574-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:40.045935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.045688 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzxr8\" (UniqueName: \"kubernetes.io/projected/678700b0-4562-45e6-ad2a-f987fac4b574-kube-api-access-rzxr8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:40.171001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.170958 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:08:40.179965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.179922 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-498mp"] Oct 30 06:08:40.217461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.217421 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:08:40.222900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.222855 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-84jsj"] Oct 30 06:08:40.263132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.263100 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:08:40.270372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:40.270291 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-387b16-openshift-default-5bb4745897-cx59q"] Oct 30 06:08:41.679723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.679679 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" path="/var/lib/kubelet/pods/3b14f741-7915-4c5f-984c-0fb0436a5959/volumes" Oct 30 06:08:41.680492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.680474 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" path="/var/lib/kubelet/pods/4571e3a6-c10d-4b7c-bdb3-2be7276a951f/volumes" Oct 30 06:08:41.681128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.681111 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" path="/var/lib/kubelet/pods/678700b0-4562-45e6-ad2a-f987fac4b574/volumes" Oct 30 06:08:41.682257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.682233 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" path="/var/lib/kubelet/pods/ac3aa7f6-fbc6-484f-9764-b25f17edb471/volumes" Oct 30 06:08:41.827805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.827759 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:41.827805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:41.827779 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:44.874750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:44.874708 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:47.280246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:47.280194 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.331862 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333333 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333355 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333368 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333378 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333427 6171 cpu_manager.go:414] "containerMap: removing container" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333435 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333462 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333470 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333484 6171 cpu_manager.go:414] "containerMap: removing container" podUID="10dd274c-e696-46e9-8da0-b7ab6e0102b7" containerName="revoke-keys" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333491 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dd274c-e696-46e9-8da0-b7ab6e0102b7" containerName="revoke-keys" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333520 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333527 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333756 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="678700b0-4562-45e6-ad2a-f987fac4b574" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333775 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac3aa7f6-fbc6-484f-9764-b25f17edb471" containerName="istio-proxy" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333793 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dd274c-e696-46e9-8da0-b7ab6e0102b7" containerName="revoke-keys" Oct 30 06:08:48.333819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333808 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.335027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333828 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b14f741-7915-4c5f-984c-0fb0436a5959" containerName="istio-proxy" Oct 30 06:08:48.335027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333859 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4571e3a6-c10d-4b7c-bdb3-2be7276a951f" containerName="istio-proxy" Oct 30 06:08:48.335027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.333872 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfde1455-eedf-4b3c-bca0-e8a9ea76df05" containerName="istio-proxy" Oct 30 06:08:48.345577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.345551 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.355894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.355868 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:08:48.363832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.363804 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d0152\"/\"kube-root-ca.crt\"" Oct 30 06:08:48.363987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.363898 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-0d0152\"/\"openshift-service-ca.crt\"" Oct 30 06:08:48.378252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.378231 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d0152\"/\"default-dockercfg-gtrwp\"" Oct 30 06:08:48.378359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.378320 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-0d0152\"/\"test-model-0d0152-kserve-self-signed-certs\"" Oct 30 06:08:48.407277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407243 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.407442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407305 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.407442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407427 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.407568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407543 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mwzz\" (UniqueName: \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.407799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407775 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.407965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.407835 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.502075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.502022 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:08:48.509570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509595 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509729 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509755 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509784 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509845 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mwzz\" (UniqueName: \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.509949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.509904 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.510154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.510118 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.510154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.510157 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.512345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.512316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.512473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.512455 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.535134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.535111 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mwzz\" (UniqueName: \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\") pod \"test-model-0d0152-kserve-fd6967d47-7g72t\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.611034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.611003 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\" (UniqueName: \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\") pod \"24543015-3a33-40ff-a581-5bd4e46b1c5b\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " Oct 30 06:08:48.611235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.611170 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\") pod \"24543015-3a33-40ff-a581-5bd4e46b1c5b\" (UID: \"24543015-3a33-40ff-a581-5bd4e46b1c5b\") " Oct 30 06:08:48.611508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.611487 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume" pod "24543015-3a33-40ff-a581-5bd4e46b1c5b" (UID: "24543015-3a33-40ff-a581-5bd4e46b1c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:48.611869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.611855 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24543015-3a33-40ff-a581-5bd4e46b1c5b-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:48.613276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.613253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5" pod "24543015-3a33-40ff-a581-5bd4e46b1c5b" (UID: "24543015-3a33-40ff-a581-5bd4e46b1c5b"). InnerVolumeSpecName "kube-api-access-tthf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:48.691067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.690306 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:48.734134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:48.728927 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tthf5\" (UniqueName: \"kubernetes.io/projected/24543015-3a33-40ff-a581-5bd4e46b1c5b-kube-api-access-tthf5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:49.124176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.124137 6171 generic.go:356] "Generic (PLEG): container finished" podID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerID="252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370" exitCode=0 Oct 30 06:08:49.124354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.124223 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" event={"ID":"24543015-3a33-40ff-a581-5bd4e46b1c5b","Type":"ContainerDied","Data":"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370"} Oct 30 06:08:49.124354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.124275 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" event={"ID":"24543015-3a33-40ff-a581-5bd4e46b1c5b","Type":"ContainerDied","Data":"51bf5d8b9257201801036fe2bfc34a6ec73af330eb149b1ead375586eecd5e0c"} Oct 30 06:08:49.124354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.124292 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr" Oct 30 06:08:49.124354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.124306 6171 scope.go:122] "RemoveContainer" containerID="252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370" Oct 30 06:08:49.156716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.155932 6171 scope.go:122] "RemoveContainer" containerID="252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370" Oct 30 06:08:49.162104 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:49.162077 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370\": container with ID starting with 252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370 not found: ID does not exist" containerID="252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370" Oct 30 06:08:49.162281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.162255 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370"} err="failed to get container status \"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370\": rpc error: code = NotFound desc = could not find container \"252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370\": container with ID starting with 252852500cda43e67467ad4bde44b4b082e1ebadb1888a8f2ee4a0fdc689e370 not found: ID does not exist" Oct 30 06:08:49.206254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.206228 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:08:49.219600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.219576 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-387b16-75454f7c9f-7wrtr"] Oct 30 06:08:49.412583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.412516 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:08:49.775008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:49.774969 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" path="/var/lib/kubelet/pods/24543015-3a33-40ff-a581-5bd4e46b1c5b/volumes" Oct 30 06:08:50.151756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:50.151629 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" event={"ID":"ca09a4e2-2323-460e-aea3-40769aab5481","Type":"ContainerStarted","Data":"883fbb34fd8f39aca5af32c5ae975537887ac3133e5caa50d5095d187b616944"} Oct 30 06:08:50.303074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:50.281176 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:08:50.328068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:50.323774 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:08:50.528848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:50.524810 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:08:50.621069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:50.614286 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:08:51.258488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:51.258448 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" event={"ID":"ca09a4e2-2323-460e-aea3-40769aab5481","Type":"ContainerStarted","Data":"4e925edb48a62638a9d477ae1bd6fe574fa674c0b61a759b9e6f364a56cdc33b"} Oct 30 06:08:51.306682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:51.306624 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" podStartSLOduration=3.306609694 podStartE2EDuration="3.306609694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:51.303286907 +0000 UTC m=+2189.743664100" watchObservedRunningTime="2026-10-30 06:08:51.306609694 +0000 UTC m=+2189.746986887" Oct 30 06:08:51.691751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:51.691678 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:51.828255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:51.828210 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:51.828469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:51.828271 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:08:52.274088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.271799 6171 generic.go:356] "Generic (PLEG): container finished" podID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerID="a1f2adebf289318c93646b2499edc347fb74da7e7b2fc6e41b819ce2131f4bfb" exitCode=0 Oct 30 06:08:52.274088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.273737 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" event={"ID":"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3","Type":"ContainerDied","Data":"a1f2adebf289318c93646b2499edc347fb74da7e7b2fc6e41b819ce2131f4bfb"} Oct 30 06:08:52.275639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.275614 6171 generic.go:356] "Generic (PLEG): container finished" podID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerID="52d8e5ed7a2321808e7de012dbc6dc112e6f12cb5c0d0add180d824406e8be8a" exitCode=0 Oct 30 06:08:52.276538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.276517 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" event={"ID":"d891f4fe-f2b4-457e-970c-049dc260f29e","Type":"ContainerDied","Data":"52d8e5ed7a2321808e7de012dbc6dc112e6f12cb5c0d0add180d824406e8be8a"} Oct 30 06:08:52.404711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.404686 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:08:52.439928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.439908 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:08:52.496460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.496427 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\" (UniqueName: \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\") pod \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " Oct 30 06:08:52.496948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.496922 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\") pod \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\" (UID: \"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3\") " Oct 30 06:08:52.497288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.497259 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\") pod \"d891f4fe-f2b4-457e-970c-049dc260f29e\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " Oct 30 06:08:52.497482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.497352 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\" (UniqueName: \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\") pod \"d891f4fe-f2b4-457e-970c-049dc260f29e\" (UID: \"d891f4fe-f2b4-457e-970c-049dc260f29e\") " Oct 30 06:08:52.501459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.501207 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq" pod "d891f4fe-f2b4-457e-970c-049dc260f29e" (UID: "d891f4fe-f2b4-457e-970c-049dc260f29e"). InnerVolumeSpecName "kube-api-access-k67rq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:52.501459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.501332 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume" pod "0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" (UID: "0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:52.501459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.501423 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume" pod "d891f4fe-f2b4-457e-970c-049dc260f29e" (UID: "d891f4fe-f2b4-457e-970c-049dc260f29e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:52.509098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.503927 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5" pod "0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" (UID: "0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3"). InnerVolumeSpecName "kube-api-access-x46m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:52.600091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.600061 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:52.600091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.600085 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d891f4fe-f2b4-457e-970c-049dc260f29e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:52.600091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.600094 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k67rq\" (UniqueName: \"kubernetes.io/projected/d891f4fe-f2b4-457e-970c-049dc260f29e-kube-api-access-k67rq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:52.600289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:52.600103 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x46m5\" (UniqueName: \"kubernetes.io/projected/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3-kube-api-access-x46m5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:08:53.286631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.286599 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" Oct 30 06:08:53.286631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.286623 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh" event={"ID":"d891f4fe-f2b4-457e-970c-049dc260f29e","Type":"ContainerDied","Data":"e923a041e02758f1d5903f8cd872f4b4fb127ff54338ca539d3e959e1cc4252c"} Oct 30 06:08:53.287220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.286684 6171 scope.go:122] "RemoveContainer" containerID="52d8e5ed7a2321808e7de012dbc6dc112e6f12cb5c0d0add180d824406e8be8a" Oct 30 06:08:53.289267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.289248 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" Oct 30 06:08:53.289383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.289293 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg" event={"ID":"0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3","Type":"ContainerDied","Data":"c967519399c1fc4d54384d11a038623e60df20c4ddbbb0a641daccfa5606e2c0"} Oct 30 06:08:53.299711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.299690 6171 scope.go:122] "RemoveContainer" containerID="a1f2adebf289318c93646b2499edc347fb74da7e7b2fc6e41b819ce2131f4bfb" Oct 30 06:08:53.347623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.347572 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:08:53.361668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.361645 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-aee070-65899ddbd9-lkdcg"] Oct 30 06:08:53.399646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.399620 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:08:53.404438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.404415 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-aee070-ff89c7dc-tbdhh"] Oct 30 06:08:53.674303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.674229 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" path="/var/lib/kubelet/pods/0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3/volumes" Oct 30 06:08:53.674677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:53.674663 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" path="/var/lib/kubelet/pods/d891f4fe-f2b4-457e-970c-049dc260f29e/volumes" Oct 30 06:08:56.434068 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:08:56.433249 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:08:58.691390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:58.691354 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:58.705774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:58.705750 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:58.759494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:58.759463 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx"] Oct 30 06:08:58.776792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:58.776757 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-3cc570-npbgx"] Oct 30 06:08:59.451691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:59.451659 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:08:59.675183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:08:59.675150 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e983376-0eb6-4bfb-8fa7-d6e4a46324a3" path="/var/lib/kubelet/pods/7e983376-0eb6-4bfb-8fa7-d6e4a46324a3/volumes" Oct 30 06:09:01.566817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.566781 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:09:01.568292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568271 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" Oct 30 06:09:01.568384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568296 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" Oct 30 06:09:01.568424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568383 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" Oct 30 06:09:01.568424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568393 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" Oct 30 06:09:01.568488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568450 6171 cpu_manager.go:414] "containerMap: removing container" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" Oct 30 06:09:01.568488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568459 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" Oct 30 06:09:01.568697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568685 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8e9f44-ae3f-439f-92f9-7c5ce96d53a3" containerName="payload-processing" Oct 30 06:09:01.568736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568710 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d891f4fe-f2b4-457e-970c-049dc260f29e" containerName="payload-pre-processing" Oct 30 06:09:01.568736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.568729 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="24543015-3a33-40ff-a581-5bd4e46b1c5b" containerName="payload-pre-processing" Oct 30 06:09:01.575596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.575569 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.599695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.599662 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:09:01.604845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.604814 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-03d349\"/\"default-dockercfg-bgbtm\"" Oct 30 06:09:01.608227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.608203 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-03d349\"/\"kube-root-ca.crt\"" Oct 30 06:09:01.608355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.608235 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-03d349\"/\"openshift-service-ca.crt\"" Oct 30 06:09:01.608355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.608333 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-03d349\"/\"auth-test-model-03d349-kserve-self-signed-certs\"" Oct 30 06:09:01.637098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637068 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qd5gt\" (UniqueName: \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.637258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637180 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.637258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.637380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637298 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.637439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637391 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.637593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.637571 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.738997 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739039 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739130 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739207 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739396 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739419 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739465 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739546 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qd5gt\" (UniqueName: \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.739705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.739667 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.741651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.741631 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.741836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.741819 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.765304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.765277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qd5gt\" (UniqueName: \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\") pod \"auth-test-model-03d349-kserve-5f5bff785c-ksqlh\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:01.910484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:01.910396 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:02.317021 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:02.316990 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52dc24b0_2ccc_4e57_aa9d_289477eff462.slice/crio-d134a4b2d221033ea5cbb9b20098fc0002ce02a73eebe59314ca017234521339 WatchSource:0}: Error finding container d134a4b2d221033ea5cbb9b20098fc0002ce02a73eebe59314ca017234521339: Status 404 returned error can't find the container with id d134a4b2d221033ea5cbb9b20098fc0002ce02a73eebe59314ca017234521339 Oct 30 06:09:02.318670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:02.318636 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:09:02.511921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:02.511882 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" event={"ID":"52dc24b0-2ccc-4e57-aa9d-289477eff462","Type":"ContainerStarted","Data":"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691"} Oct 30 06:09:02.512177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:02.511937 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" event={"ID":"52dc24b0-2ccc-4e57-aa9d-289477eff462","Type":"ContainerStarted","Data":"d134a4b2d221033ea5cbb9b20098fc0002ce02a73eebe59314ca017234521339"} Oct 30 06:09:02.560803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:02.560728 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" podStartSLOduration=1.560705863 podStartE2EDuration="1.560705863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:02.556965855 +0000 UTC m=+2200.997343048" watchObservedRunningTime="2026-10-30 06:09:02.560705863 +0000 UTC m=+2201.001083059" Oct 30 06:09:02.911769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:02.911722 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:06.235072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.234647 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r"] Oct 30 06:09:06.284521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.243637 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-aee070-48g9r"] Oct 30 06:09:06.349669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.349626 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:09:06.350792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.350573 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" containerID="cri-o://9e0e8c0aae924be81827c7d601faddc15ea1def5d8848eac10afe46789a4dd52" gracePeriod=30 Oct 30 06:09:06.854175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.854132 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx"] Oct 30 06:09:06.873686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.871513 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:06.878103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.877878 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx"] Oct 30 06:09:06.891681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.891636 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:09:06.891890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:06.891715 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:07.031172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.031131 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98vbb\" (UniqueName: \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.031389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.031260 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.031505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.031488 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.132731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.132637 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98vbb\" (UniqueName: \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.132731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.132700 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.132995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.132881 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.133617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.133585 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.135608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.135580 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.167608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.159409 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98vbb\" (UniqueName: \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\") pod \"maas-api-revoke-sub-680f1658-h6tpx\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.202235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.202204 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:07.524846 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:07.524808 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad879bdb_cd1f_442d_ae9f_ca72283301ee.slice/crio-52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde WatchSource:0}: Error finding container 52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde: Status 404 returned error can't find the container with id 52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde Oct 30 06:09:07.527267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.527110 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx"] Oct 30 06:09:07.677246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.677206 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="726eaea3-4d9c-424e-bf8f-48c0d5469ea0" path="/var/lib/kubelet/pods/726eaea3-4d9c-424e-bf8f-48c0d5469ea0/volumes" Oct 30 06:09:07.878623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:07.878582 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" event={"ID":"ad879bdb-cd1f-442d-ae9f-ca72283301ee","Type":"ContainerStarted","Data":"52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde"} Oct 30 06:09:08.892461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:08.892357 6171 generic.go:356] "Generic (PLEG): container finished" podID="ad879bdb-cd1f-442d-ae9f-ca72283301ee" containerID="a6804f8759ac0a1bdd6ed185949e6363a256e7eaf440104c8f3185f170557297" exitCode=0 Oct 30 06:09:08.892461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:08.892447 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" event={"ID":"ad879bdb-cd1f-442d-ae9f-ca72283301ee","Type":"ContainerDied","Data":"a6804f8759ac0a1bdd6ed185949e6363a256e7eaf440104c8f3185f170557297"} Oct 30 06:09:10.103531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.103501 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:10.193205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.193102 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\") pod \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " Oct 30 06:09:10.193205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.193185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\" (UniqueName: \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\") pod \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " Oct 30 06:09:10.193611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.193577 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca" pod "ad879bdb-cd1f-442d-ae9f-ca72283301ee" (UID: "ad879bdb-cd1f-442d-ae9f-ca72283301ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:10.193689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.193585 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\") pod \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\" (UID: \"ad879bdb-cd1f-442d-ae9f-ca72283301ee\") " Oct 30 06:09:10.195742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.195703 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:10.195896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.195817 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca" pod "ad879bdb-cd1f-442d-ae9f-ca72283301ee" (UID: "ad879bdb-cd1f-442d-ae9f-ca72283301ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:10.195896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.195875 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb" pod "ad879bdb-cd1f-442d-ae9f-ca72283301ee" (UID: "ad879bdb-cd1f-442d-ae9f-ca72283301ee"). InnerVolumeSpecName "kube-api-access-98vbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:10.297105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.297062 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98vbb\" (UniqueName: \"kubernetes.io/projected/ad879bdb-cd1f-442d-ae9f-ca72283301ee-kube-api-access-98vbb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:10.297105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.297098 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad879bdb-cd1f-442d-ae9f-ca72283301ee-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:10.640022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.639982 6171 patch_prober.go:29] interesting pod/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:10.640236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.640061 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:10.918063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.917970 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" Oct 30 06:09:10.918063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.917981 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx" event={"ID":"ad879bdb-cd1f-442d-ae9f-ca72283301ee","Type":"ContainerDied","Data":"52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde"} Oct 30 06:09:10.918063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:10.918027 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52778fbac101d7f40d0f37c60a471b46997641bd610abe16dfffcdc96777dbde" Oct 30 06:09:11.892670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.892630 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:09:11.893591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.893575 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ad879bdb-cd1f-442d-ae9f-ca72283301ee" containerName="revoke-keys" Oct 30 06:09:11.893668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.893594 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad879bdb-cd1f-442d-ae9f-ca72283301ee" containerName="revoke-keys" Oct 30 06:09:11.893774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.893763 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad879bdb-cd1f-442d-ae9f-ca72283301ee" containerName="revoke-keys" Oct 30 06:09:11.897970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.897950 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.912596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.912567 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:11.916260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.916227 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:09:11.922457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.922423 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-cfe14f-openshift-default-dockercfg-hbfwm\"" Oct 30 06:09:11.931821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.931711 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932089 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932128 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932361 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932513 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932654 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qts8q\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932766 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.932912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.932895 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.933418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.933084 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:11.935906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.935876 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:11.944371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.944295 6171 generic.go:356] "Generic (PLEG): container finished" podID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerID="9e0e8c0aae924be81827c7d601faddc15ea1def5d8848eac10afe46789a4dd52" exitCode=0 Oct 30 06:09:11.946087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.945987 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" event={"ID":"fbb7c37a-6b95-4370-979e-3d8f261ae7f3","Type":"ContainerDied","Data":"9e0e8c0aae924be81827c7d601faddc15ea1def5d8848eac10afe46789a4dd52"} Oct 30 06:09:11.966561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:11.964138 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:09:12.035140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035038 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qts8q\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035199 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035274 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035371 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035497 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035626 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035661 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035748 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035777 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.035938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.035916 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.036362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.036331 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.037101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.037072 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.037221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.037196 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.037396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.037374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.038924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.038900 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.040227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.040183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.065089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.065003 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qts8q\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.068387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.068359 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\") pod \"e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.134640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.134573 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:09:12.220012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.219974 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:12.239911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.239881 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.239924 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.239994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240175 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240237 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240292 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240372 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240406 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240551 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\") pod \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\" (UID: \"fbb7c37a-6b95-4370-979e-3d8f261ae7f3\") " Oct 30 06:09:12.240628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240364 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:12.240628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240600 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:12.240965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.240941 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:12.241363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241330 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:12.241525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241506 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:12.241714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241690 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.241714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241714 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.241903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241727 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.241903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241736 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.241903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.241744 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.243847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.243821 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:12.243984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.243855 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "kube-api-access-5g4df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:12.243984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.243921 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:12.244992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.244969 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy" pod "fbb7c37a-6b95-4370-979e-3d8f261ae7f3" (UID: "fbb7c37a-6b95-4370-979e-3d8f261ae7f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:12.343791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.343752 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.343791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.343789 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g4df\" (UniqueName: \"kubernetes.io/projected/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-kube-api-access-5g4df\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.344033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.343800 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.344033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.343809 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fbb7c37a-6b95-4370-979e-3d8f261ae7f3-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.561955 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:12.561919 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c881bc_70da_4e83_be3b_d20fee1b1ae9.slice/crio-c74d3a16352f21ccf4cdef178853cf47f0fbc4b129e0e2a11de01de8f1ab8d36 WatchSource:0}: Error finding container c74d3a16352f21ccf4cdef178853cf47f0fbc4b129e0e2a11de01de8f1ab8d36: Status 404 returned error can't find the container with id c74d3a16352f21ccf4cdef178853cf47f0fbc4b129e0e2a11de01de8f1ab8d36 Oct 30 06:09:12.563676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.563650 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:09:12.565651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.565614 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:12.565721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.565687 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:12.565721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.565717 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:12.967457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.967370 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" Oct 30 06:09:12.967457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.967396 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn" event={"ID":"fbb7c37a-6b95-4370-979e-3d8f261ae7f3","Type":"ContainerDied","Data":"c3c964c638ff515842c3e90f2f9ba659939049fe1617820f264d54f0619611ef"} Oct 30 06:09:12.968091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.967468 6171 scope.go:122] "RemoveContainer" containerID="9e0e8c0aae924be81827c7d601faddc15ea1def5d8848eac10afe46789a4dd52" Oct 30 06:09:12.970088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.970054 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" event={"ID":"e3c881bc-70da-4e83-be3b-d20fee1b1ae9","Type":"ContainerStarted","Data":"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e"} Oct 30 06:09:12.970237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:12.970124 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" event={"ID":"e3c881bc-70da-4e83-be3b-d20fee1b1ae9","Type":"ContainerStarted","Data":"c74d3a16352f21ccf4cdef178853cf47f0fbc4b129e0e2a11de01de8f1ab8d36"} Oct 30 06:09:13.036610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.036574 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:09:13.048452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.048421 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-3cc570-openshift-default-7f6b9c66ff-44crn"] Oct 30 06:09:13.098947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.098884 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podStartSLOduration=2.09887011 podStartE2EDuration="2.09887011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:13.094872912 +0000 UTC m=+2211.535250105" watchObservedRunningTime="2026-10-30 06:09:13.09887011 +0000 UTC m=+2211.539247303" Oct 30 06:09:13.221356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.221268 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:13.222912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.222871 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:13.223109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.222936 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:13.675618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:13.675510 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" path="/var/lib/kubelet/pods/fbb7c37a-6b95-4370-979e-3d8f261ae7f3/volumes" Oct 30 06:09:14.149138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.149101 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:09:14.149708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.149380 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" containerID="cri-o://2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664" gracePeriod=30 Oct 30 06:09:14.226196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.226150 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:14.226411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.226220 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:14.684476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.684398 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88"] Oct 30 06:09:14.686359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.686175 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" Oct 30 06:09:14.686565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.686545 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" Oct 30 06:09:14.687080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.687060 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbb7c37a-6b95-4370-979e-3d8f261ae7f3" containerName="istio-proxy" Oct 30 06:09:14.792037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.791979 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:14.833530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.833490 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:09:14.846370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.846335 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88"] Oct 30 06:09:14.895372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.895331 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:14.912781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.912746 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:14.913008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.912807 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q4rn\" (UniqueName: \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:14.913180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:14.912999 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.014403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.014354 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.014616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.014455 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q4rn\" (UniqueName: \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.014616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.014586 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.015393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.015357 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.017336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.017311 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.040201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.040169 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q4rn\" (UniqueName: \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\") pod \"maas-api-revoke-keys-e2e-inf-b-c54211-84h88\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.152689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.152659 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:15.221213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.221171 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:15.221401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.221237 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:15.327513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.325930 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8"] Oct 30 06:09:15.332681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.332315 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8"] Oct 30 06:09:15.332681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.332448 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.423251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.423212 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.423612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.423575 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rhrj\" (UniqueName: \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.423934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.423911 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.515855 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:15.515823 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e72a7e_08c2_4deb_aad9_de6acb63cd9e.slice/crio-29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391 WatchSource:0}: Error finding container 29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391: Status 404 returned error can't find the container with id 29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391 Oct 30 06:09:15.517027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.517004 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88"] Oct 30 06:09:15.525252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.525221 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.525377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.525311 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rhrj\" (UniqueName: \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.525528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.525513 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.526390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.526366 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.528383 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.551443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.551421 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rhrj\" (UniqueName: \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\") pod \"maas-api-revoke-sub-5e8c249a-f96w8\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:15.674428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:15.674349 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:16.018111 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:16.018076 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e8a02e8_eeb5_4aed_910e_b52f4ed67464.slice/crio-8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1 WatchSource:0}: Error finding container 8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1: Status 404 returned error can't find the container with id 8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1 Oct 30 06:09:16.019277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.019255 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8"] Oct 30 06:09:16.036236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.036201 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" event={"ID":"1e8a02e8-eeb5-4aed-910e-b52f4ed67464","Type":"ContainerStarted","Data":"8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1"} Oct 30 06:09:16.038568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.038540 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" event={"ID":"54e72a7e-08c2-4deb-aad9-de6acb63cd9e","Type":"ContainerStarted","Data":"93de8a5ef3fd79be11bc91530fa9fc20941469d18f1d085771830ad95708295d"} Oct 30 06:09:16.038728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.038591 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" event={"ID":"54e72a7e-08c2-4deb-aad9-de6acb63cd9e","Type":"ContainerStarted","Data":"29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391"} Oct 30 06:09:16.104070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.102382 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" podStartSLOduration=2.102360481 podStartE2EDuration="2.102360481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:16.101271724 +0000 UTC m=+2214.541648918" watchObservedRunningTime="2026-10-30 06:09:16.102360481 +0000 UTC m=+2214.542737676" Oct 30 06:09:16.222936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.222899 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:16.223580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:16.223545 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:17.049781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.049738 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" event={"ID":"1e8a02e8-eeb5-4aed-910e-b52f4ed67464","Type":"ContainerStarted","Data":"0187e5cca985f92e5f49c3fb02143f1356d7a3349d3e7b80d5157e253997dc4c"} Oct 30 06:09:17.051531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.051504 6171 generic.go:356] "Generic (PLEG): container finished" podID="54e72a7e-08c2-4deb-aad9-de6acb63cd9e" containerID="93de8a5ef3fd79be11bc91530fa9fc20941469d18f1d085771830ad95708295d" exitCode=0 Oct 30 06:09:17.051672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.051579 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" event={"ID":"54e72a7e-08c2-4deb-aad9-de6acb63cd9e","Type":"ContainerDied","Data":"93de8a5ef3fd79be11bc91530fa9fc20941469d18f1d085771830ad95708295d"} Oct 30 06:09:17.108464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.108388 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" podStartSLOduration=2.108371785 podStartE2EDuration="2.108371785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:17.105874171 +0000 UTC m=+2215.546251365" watchObservedRunningTime="2026-10-30 06:09:17.108371785 +0000 UTC m=+2215.548748979" Oct 30 06:09:17.220868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.220832 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:17.221090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:17.220887 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:18.068079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.068026 6171 generic.go:356] "Generic (PLEG): container finished" podID="1e8a02e8-eeb5-4aed-910e-b52f4ed67464" containerID="0187e5cca985f92e5f49c3fb02143f1356d7a3349d3e7b80d5157e253997dc4c" exitCode=0 Oct 30 06:09:18.068516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.068120 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" event={"ID":"1e8a02e8-eeb5-4aed-910e-b52f4ed67464","Type":"ContainerDied","Data":"0187e5cca985f92e5f49c3fb02143f1356d7a3349d3e7b80d5157e253997dc4c"} Oct 30 06:09:18.223942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.223500 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:18.223942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.223586 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:18.377267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.377239 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:18.510070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.507869 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\" (UniqueName: \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\") pod \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " Oct 30 06:09:18.510070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.508450 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\") pod \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " Oct 30 06:09:18.510070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.508828 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\") pod \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\" (UID: \"54e72a7e-08c2-4deb-aad9-de6acb63cd9e\") " Oct 30 06:09:18.510070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.509328 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca" pod "54e72a7e-08c2-4deb-aad9-de6acb63cd9e" (UID: "54e72a7e-08c2-4deb-aad9-de6acb63cd9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:18.510070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.509829 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:18.511502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.511471 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn" pod "54e72a7e-08c2-4deb-aad9-de6acb63cd9e" (UID: "54e72a7e-08c2-4deb-aad9-de6acb63cd9e"). InnerVolumeSpecName "kube-api-access-8q4rn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:18.511728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.511699 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca" pod "54e72a7e-08c2-4deb-aad9-de6acb63cd9e" (UID: "54e72a7e-08c2-4deb-aad9-de6acb63cd9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:18.611813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.611726 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:18.611813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:18.611758 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q4rn\" (UniqueName: \"kubernetes.io/projected/54e72a7e-08c2-4deb-aad9-de6acb63cd9e-kube-api-access-8q4rn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.080021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.079985 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" Oct 30 06:09:19.080534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.080017 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88" event={"ID":"54e72a7e-08c2-4deb-aad9-de6acb63cd9e","Type":"ContainerDied","Data":"29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391"} Oct 30 06:09:19.080534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.080111 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29fa5d9088717f9a71ebf280418fa5f8201f04f9cba5b3e5281493d92cec9391" Oct 30 06:09:19.222222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.221870 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" start-of-body= Oct 30 06:09:19.222222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.221916 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.32:15021/healthz/ready\": dial tcp 10.133.1.32:15021: connect: connection refused" Oct 30 06:09:19.328552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.328525 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:19.427574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.426943 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\") pod \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " Oct 30 06:09:19.427574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.426977 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\") pod \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " Oct 30 06:09:19.427574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.427427 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca" pod "1e8a02e8-eeb5-4aed-910e-b52f4ed67464" (UID: "1e8a02e8-eeb5-4aed-910e-b52f4ed67464"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:19.427574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.427534 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\" (UniqueName: \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\") pod \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\" (UID: \"1e8a02e8-eeb5-4aed-910e-b52f4ed67464\") " Oct 30 06:09:19.428426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.428392 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.431032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.430991 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca" pod "1e8a02e8-eeb5-4aed-910e-b52f4ed67464" (UID: "1e8a02e8-eeb5-4aed-910e-b52f4ed67464"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:19.432688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.432664 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj" pod "1e8a02e8-eeb5-4aed-910e-b52f4ed67464" (UID: "1e8a02e8-eeb5-4aed-910e-b52f4ed67464"). InnerVolumeSpecName "kube-api-access-4rhrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.565550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.564390 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.565550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.564432 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rhrj\" (UniqueName: \"kubernetes.io/projected/1e8a02e8-eeb5-4aed-910e-b52f4ed67464-kube-api-access-4rhrj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.660708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.660675 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:09:19.767018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.766985 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767025 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767084 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767154 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767191 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767307 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767352 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767405 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.767688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767667 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\") pod \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\" (UID: \"4a4e56d4-52c3-495c-b7b0-cdef32b909a0\") " Oct 30 06:09:19.768754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767404 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:19.768754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767636 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.768754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767817 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.768754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.767962 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.769122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.768789 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.769319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.769293 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.769417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.769325 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.769417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.769341 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.769417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.769353 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.769417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.769366 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.770381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.770353 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "kube-api-access-cwnfd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.770476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.770455 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:19.770628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.770610 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:19.771484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.771471 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo" pod "4a4e56d4-52c3-495c-b7b0-cdef32b909a0" (UID: "4a4e56d4-52c3-495c-b7b0-cdef32b909a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:19.870804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.870772 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.870804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.870802 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwnfd\" (UniqueName: \"kubernetes.io/projected/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-kube-api-access-cwnfd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.870804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.870812 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:19.871128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:19.870825 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a4e56d4-52c3-495c-b7b0-cdef32b909a0-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:20.094308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.094274 6171 generic.go:356] "Generic (PLEG): container finished" podID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerID="2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664" exitCode=0 Oct 30 06:09:20.094807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.094328 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" Oct 30 06:09:20.094807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.094371 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerDied","Data":"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664"} Oct 30 06:09:20.094807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.094437 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5" event={"ID":"4a4e56d4-52c3-495c-b7b0-cdef32b909a0","Type":"ContainerDied","Data":"d134c67db231a348b9654b59251faf658bab8d043741e071133fe313fa630e5e"} Oct 30 06:09:20.094807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.094473 6171 scope.go:122] "RemoveContainer" containerID="2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664" Oct 30 06:09:20.096290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.096201 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" Oct 30 06:09:20.096290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.096204 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8" event={"ID":"1e8a02e8-eeb5-4aed-910e-b52f4ed67464","Type":"ContainerDied","Data":"8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1"} Oct 30 06:09:20.096290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.096250 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ea27d7fcf5d9dc8a25a30eefd509f905f411a95cc774cfb53b4975b3eba20c1" Oct 30 06:09:20.109224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.109014 6171 scope.go:122] "RemoveContainer" containerID="4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657" Oct 30 06:09:20.120870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.120840 6171 scope.go:122] "RemoveContainer" containerID="2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664" Oct 30 06:09:20.121243 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:20.121217 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664\": container with ID starting with 2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664 not found: ID does not exist" containerID="2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664" Oct 30 06:09:20.121358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.121252 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664"} err="failed to get container status \"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664\": rpc error: code = NotFound desc = could not find container \"2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664\": container with ID starting with 2c6b0f263632a2cb3d9ab38051bb3bab0d0a604f843110dd49b7654d62718664 not found: ID does not exist" Oct 30 06:09:20.121358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.121273 6171 scope.go:122] "RemoveContainer" containerID="4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657" Oct 30 06:09:20.121600 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:20.121582 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657\": container with ID starting with 4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657 not found: ID does not exist" containerID="4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657" Oct 30 06:09:20.121669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.121608 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657"} err="failed to get container status \"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657\": rpc error: code = NotFound desc = could not find container \"4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657\": container with ID starting with 4f5d6172025e9ff860b84e8e4ffaf76996735a701c35a1ca27274cdf2d587657 not found: ID does not exist" Oct 30 06:09:20.175434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.175407 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:09:20.180895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.180876 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-aee070-openshift-default-5b997d785f-vmnp5"] Oct 30 06:09:20.473235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.473198 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:20.474400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.474137 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:20.477172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:20.477148 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:09:21.573434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:21.573400 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b"] Oct 30 06:09:21.579871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:21.579844 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-7ad9ad-dwj8b"] Oct 30 06:09:21.724065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:21.723960 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dd274c-e696-46e9-8da0-b7ab6e0102b7" path="/var/lib/kubelet/pods/10dd274c-e696-46e9-8da0-b7ab6e0102b7/volumes" Oct 30 06:09:21.725274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:21.725243 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" path="/var/lib/kubelet/pods/4a4e56d4-52c3-495c-b7b0-cdef32b909a0/volumes" Oct 30 06:09:25.085415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:25.085376 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:09:25.113361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:25.113326 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:09:26.384137 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:26.384095 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:09:27.052069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:27.049785 6171 scope.go:122] "RemoveContainer" containerID="c50b16cd379bf980ff93cbd7e202fe48abd3bdd0af70592145b4186bd13dd717" Oct 30 06:09:27.069910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:27.069887 6171 scope.go:122] "RemoveContainer" containerID="bb6945bee11572514244c09d01dddea9cb97f080b951a5ba2cb1fb2f3b159cd6" Oct 30 06:09:29.466407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:29.466374 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:09:29.466873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:29.466580 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" containerID="cri-o://6a9c3f5a5f9016b06e4766b1c51419f8b160766a4534d529eb53784b1d3d090d" gracePeriod=30 Oct 30 06:09:30.086739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.086644 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" containerID="cri-o://6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae" gracePeriod=25 Oct 30 06:09:30.114009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.113971 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" containerID="cri-o://8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d" gracePeriod=25 Oct 30 06:09:30.746868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.746821 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:09:30.748275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748248 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:30.748275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748275 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748287 6171 cpu_manager.go:414] "containerMap: removing container" podUID="54e72a7e-08c2-4deb-aad9-de6acb63cd9e" containerName="revoke-keys" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748296 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54e72a7e-08c2-4deb-aad9-de6acb63cd9e" containerName="revoke-keys" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748330 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748347 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748358 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1e8a02e8-eeb5-4aed-910e-b52f4ed67464" containerName="revoke-keys" Oct 30 06:09:30.748507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748365 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e8a02e8-eeb5-4aed-910e-b52f4ed67464" containerName="revoke-keys" Oct 30 06:09:30.748820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748581 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:30.748820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748597 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e8a02e8-eeb5-4aed-910e-b52f4ed67464" containerName="revoke-keys" Oct 30 06:09:30.748820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.748629 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="54e72a7e-08c2-4deb-aad9-de6acb63cd9e" containerName="revoke-keys" Oct 30 06:09:30.756027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.756000 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.772808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.772784 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:09:30.779801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.779769 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-d1bfce-openshift-default-dockercfg-xxfmp\"" Oct 30 06:09:30.877206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877176 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877227 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877250 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877384 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877420 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877629 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877683 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877764 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.877827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.877810 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j5z7\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987269 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987339 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987386 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987443 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j5z7\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987662 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987694 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987747 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:30.989070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:30.987773 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.000595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.000517 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.001115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.001032 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.002190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.001536 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.002190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.001820 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.012074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.003131 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.034065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.033027 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.139069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.135601 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.191066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.188006 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j5z7\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.279260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.271862 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\") pod \"e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.316564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.316528 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:09:31.374755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.374711 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:31.624625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.623719 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:31.851201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.851121 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc"] Oct 30 06:09:31.852463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.852442 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a4e56d4-52c3-495c-b7b0-cdef32b909a0" containerName="istio-proxy" Oct 30 06:09:31.857890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.857869 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.877842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.877269 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:31.877842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.877449 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:09:31.878504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.878477 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc"] Oct 30 06:09:31.880017 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:31.879983 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3dbdee0_5c5c_4fc3_a5de_0d0d319f5967.slice/crio-26b2609c7185ca4ae639f1480a775ddddefbded2db23f918e2e0a36dd22e79a9 WatchSource:0}: Error finding container 26b2609c7185ca4ae639f1480a775ddddefbded2db23f918e2e0a36dd22e79a9: Status 404 returned error can't find the container with id 26b2609c7185ca4ae639f1480a775ddddefbded2db23f918e2e0a36dd22e79a9 Oct 30 06:09:31.884468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.884428 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:31.884591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.884519 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:31.884591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.884566 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:31.886649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.886573 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:09:31.890026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.888878 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.890026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.889011 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7m5m\" (UniqueName: \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.890026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.889333 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.990904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.990869 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.991085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.990915 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.991164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.991086 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7m5m\" (UniqueName: \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.991686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.991664 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:31.993480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:31.993457 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:32.016528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.016499 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7m5m\" (UniqueName: \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\") pod \"maas-api-revoke-sub-a2d9b776-jmtdc\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:32.178426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.178334 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:32.583233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.583197 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc"] Oct 30 06:09:32.586179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.586146 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerStarted","Data":"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8"} Oct 30 06:09:32.586398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.586376 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerStarted","Data":"26b2609c7185ca4ae639f1480a775ddddefbded2db23f918e2e0a36dd22e79a9"} Oct 30 06:09:32.638114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.638067 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podStartSLOduration=2.638029953 podStartE2EDuration="2.638029953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:32.635869475 +0000 UTC m=+2231.076246668" watchObservedRunningTime="2026-10-30 06:09:32.638029953 +0000 UTC m=+2231.078407147" Oct 30 06:09:32.886038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:32.886005 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:33.375296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.375263 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:33.381898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.381858 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:33.599663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.599631 6171 generic.go:356] "Generic (PLEG): container finished" podID="aa7d5ccb-f516-4154-af11-c258c78c20e9" containerID="7810f340ec95a750bb979e5e7aa3cdf2e29de830aa2ebe706996f2a8b91cccb2" exitCode=0 Oct 30 06:09:33.599826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.599722 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" event={"ID":"aa7d5ccb-f516-4154-af11-c258c78c20e9","Type":"ContainerDied","Data":"7810f340ec95a750bb979e5e7aa3cdf2e29de830aa2ebe706996f2a8b91cccb2"} Oct 30 06:09:33.599826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.599779 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" event={"ID":"aa7d5ccb-f516-4154-af11-c258c78c20e9","Type":"ContainerStarted","Data":"ccc4291d5a5decac7fd959194e763bc3c6c4279121f12487d25da4bab86d7462"} Oct 30 06:09:33.599978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.599932 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:33.601221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:33.601121 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:09:34.611003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.610974 6171 generic.go:356] "Generic (PLEG): container finished" podID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerID="6a9c3f5a5f9016b06e4766b1c51419f8b160766a4534d529eb53784b1d3d090d" exitCode=0 Oct 30 06:09:34.611440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.611004 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" event={"ID":"7913728c-2dbb-4b30-bf7e-ae6c70211426","Type":"ContainerDied","Data":"6a9c3f5a5f9016b06e4766b1c51419f8b160766a4534d529eb53784b1d3d090d"} Oct 30 06:09:34.789890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.789856 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:09:34.825182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825152 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.825366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825214 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.825455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825369 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.825517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825468 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.825656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825621 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.825737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825685 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.826205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825701 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.826205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.825724 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.826205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.826183 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.827921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.826289 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:34.827921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.826578 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.827921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.826813 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.827921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.826872 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\") pod \"7913728c-2dbb-4b30-bf7e-ae6c70211426\" (UID: \"7913728c-2dbb-4b30-bf7e-ae6c70211426\") " Oct 30 06:09:34.827921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.827118 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.828303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828119 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.828303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828143 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7913728c-2dbb-4b30-bf7e-ae6c70211426-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.828303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828158 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.828303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828172 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.828303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828189 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.828722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828700 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.828783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828746 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:34.828853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.828779 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "kube-api-access-h5ljk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.829152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.829132 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy" pod "7913728c-2dbb-4b30-bf7e-ae6c70211426" (UID: "7913728c-2dbb-4b30-bf7e-ae6c70211426"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:34.881198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.881162 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:34.929195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.929156 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\") pod \"aa7d5ccb-f516-4154-af11-c258c78c20e9\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " Oct 30 06:09:34.929688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.929280 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\" (UniqueName: \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\") pod \"aa7d5ccb-f516-4154-af11-c258c78c20e9\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " Oct 30 06:09:34.929688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.929504 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\") pod \"aa7d5ccb-f516-4154-af11-c258c78c20e9\" (UID: \"aa7d5ccb-f516-4154-af11-c258c78c20e9\") " Oct 30 06:09:34.929992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.929959 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca" pod "aa7d5ccb-f516-4154-af11-c258c78c20e9" (UID: "aa7d5ccb-f516-4154-af11-c258c78c20e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:34.930439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.930416 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.930546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.930443 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.930546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.930458 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h5ljk\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-kube-api-access-h5ljk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.930546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.930472 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.930546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.930484 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7913728c-2dbb-4b30-bf7e-ae6c70211426-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:34.932223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.932195 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m" pod "aa7d5ccb-f516-4154-af11-c258c78c20e9" (UID: "aa7d5ccb-f516-4154-af11-c258c78c20e9"). InnerVolumeSpecName "kube-api-access-f7m5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:34.933104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:34.933030 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca" pod "aa7d5ccb-f516-4154-af11-c258c78c20e9" (UID: "aa7d5ccb-f516-4154-af11-c258c78c20e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:35.032590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.032512 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa7d5ccb-f516-4154-af11-c258c78c20e9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:35.032590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.032544 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7m5m\" (UniqueName: \"kubernetes.io/projected/aa7d5ccb-f516-4154-af11-c258c78c20e9-kube-api-access-f7m5m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:35.623062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.623019 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" Oct 30 06:09:35.623537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.623087 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt" event={"ID":"7913728c-2dbb-4b30-bf7e-ae6c70211426","Type":"ContainerDied","Data":"0baf19237e8e1814cfbb848422af163abc8e7d11dd0c55c2917ce77893e5ab19"} Oct 30 06:09:35.623537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.623161 6171 scope.go:122] "RemoveContainer" containerID="6a9c3f5a5f9016b06e4766b1c51419f8b160766a4534d529eb53784b1d3d090d" Oct 30 06:09:35.625500 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.625476 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" Oct 30 06:09:35.625644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.625504 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc" event={"ID":"aa7d5ccb-f516-4154-af11-c258c78c20e9","Type":"ContainerDied","Data":"ccc4291d5a5decac7fd959194e763bc3c6c4279121f12487d25da4bab86d7462"} Oct 30 06:09:35.625644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.625542 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ccc4291d5a5decac7fd959194e763bc3c6c4279121f12487d25da4bab86d7462" Oct 30 06:09:35.763830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.763755 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:09:35.777655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:35.777618 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7ad9ad-openshift-default-78bdfc7bd9-rjzmt"] Oct 30 06:09:36.716060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.716020 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:09:36.717361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717325 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" Oct 30 06:09:36.717361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717355 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" Oct 30 06:09:36.717841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717442 6171 cpu_manager.go:414] "containerMap: removing container" podUID="aa7d5ccb-f516-4154-af11-c258c78c20e9" containerName="revoke-keys" Oct 30 06:09:36.717841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717452 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa7d5ccb-f516-4154-af11-c258c78c20e9" containerName="revoke-keys" Oct 30 06:09:36.717841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717642 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" containerName="istio-proxy" Oct 30 06:09:36.717841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.717673 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa7d5ccb-f516-4154-af11-c258c78c20e9" containerName="revoke-keys" Oct 30 06:09:36.723191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.723171 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:36.739943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.739913 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-d1bfce-dockercfg-fsmjv\"" Oct 30 06:09:36.740401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.740381 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-d1bfce\"" Oct 30 06:09:36.742289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.742269 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:09:36.764598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.764541 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:36.764880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.764855 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6fzq\" (UniqueName: \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:36.778635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.778592 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:09:36.784462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.784435 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:36.802987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.802960 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:09:36.974496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.974460 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6fzq\" (UniqueName: \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:36.975034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.975006 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:36.982069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:36.976364 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:37.128884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.128847 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6fzq\" (UniqueName: \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\") pod \"payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:37.130069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.130019 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2f2z\" (UniqueName: \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.130431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.130411 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.232243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.232152 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2f2z\" (UniqueName: \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.232438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.232259 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.232998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.232978 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.323071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.298858 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2f2z\" (UniqueName: \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\") pod \"payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.357389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.338691 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:37.420097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.416371 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:37.675495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.675459 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7913728c-2dbb-4b30-bf7e-ae6c70211426" path="/var/lib/kubelet/pods/7913728c-2dbb-4b30-bf7e-ae6c70211426/volumes" Oct 30 06:09:37.796357 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:37.796312 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefd2020_07b8_49de_9b70_61b8afc90036.slice/crio-3f7d4b98aa5c4ecad20dc1e4b7ef24b8fde0dd6c0c5dd595a0278761c84a87db WatchSource:0}: Error finding container 3f7d4b98aa5c4ecad20dc1e4b7ef24b8fde0dd6c0c5dd595a0278761c84a87db: Status 404 returned error can't find the container with id 3f7d4b98aa5c4ecad20dc1e4b7ef24b8fde0dd6c0c5dd595a0278761c84a87db Oct 30 06:09:37.796873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.796833 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:09:37.808014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:37.807986 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:09:38.663856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.663820 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" event={"ID":"4d67f174-f80a-452d-ae16-384c4f5ab28f","Type":"ContainerStarted","Data":"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df"} Oct 30 06:09:38.664099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.663871 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" event={"ID":"4d67f174-f80a-452d-ae16-384c4f5ab28f","Type":"ContainerStarted","Data":"5ef1712e80249d5c1fc568cfaa939837496680fbd9968dc795a66ac167a0518a"} Oct 30 06:09:38.664099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.663898 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:38.666123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.666088 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" event={"ID":"fefd2020-07b8-49de-9b70-61b8afc90036","Type":"ContainerStarted","Data":"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb"} Oct 30 06:09:38.666286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.666140 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" event={"ID":"fefd2020-07b8-49de-9b70-61b8afc90036","Type":"ContainerStarted","Data":"3f7d4b98aa5c4ecad20dc1e4b7ef24b8fde0dd6c0c5dd595a0278761c84a87db"} Oct 30 06:09:38.666286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.666216 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:38.713558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.713508 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" podStartSLOduration=2.713494126 podStartE2EDuration="2.713494126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:38.708232359 +0000 UTC m=+2237.148609551" watchObservedRunningTime="2026-10-30 06:09:38.713494126 +0000 UTC m=+2237.153871319" Oct 30 06:09:38.756889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:38.756831 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" podStartSLOduration=2.756813825 podStartE2EDuration="2.756813825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:38.75148372 +0000 UTC m=+2237.191860912" watchObservedRunningTime="2026-10-30 06:09:38.756813825 +0000 UTC m=+2237.197191017" Oct 30 06:09:41.616947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:41.616908 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:42.885317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:42.885274 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:09:43.918006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:43.917964 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:09:44.485202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.485170 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f"] Oct 30 06:09:44.490646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.490624 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.508690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.508664 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f"] Oct 30 06:09:44.516307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.516248 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:44.516513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.516494 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:09:44.523398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.523367 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tsst\" (UniqueName: \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.523553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.523534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.523764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.523739 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.625608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.625572 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.625815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.625698 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tsst\" (UniqueName: \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.625815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.625798 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.626422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.626393 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.628426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.628399 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.651105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.651079 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tsst\" (UniqueName: \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\") pod \"maas-api-revoke-sub-7ee8e44e-5d72f\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:44.809126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:44.809020 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:45.123200 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:45.123166 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c1ead80_a52a_4bd4_80fc_d815712b7b40.slice/crio-9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4 WatchSource:0}: Error finding container 9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4: Status 404 returned error can't find the container with id 9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4 Oct 30 06:09:45.124121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:45.124094 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f"] Oct 30 06:09:45.755837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:45.755757 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" event={"ID":"2c1ead80-a52a-4bd4-80fc-d815712b7b40","Type":"ContainerStarted","Data":"bd6ab2d58b9e2d5bd7f9af05a8f7128de7b0a25948b31048287d77f81962b712"} Oct 30 06:09:45.755837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:45.755809 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" event={"ID":"2c1ead80-a52a-4bd4-80fc-d815712b7b40","Type":"ContainerStarted","Data":"9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4"} Oct 30 06:09:45.801539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:45.801475 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" podStartSLOduration=1.801457762 podStartE2EDuration="1.801457762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:45.799349704 +0000 UTC m=+2244.239726899" watchObservedRunningTime="2026-10-30 06:09:45.801457762 +0000 UTC m=+2244.241834957" Oct 30 06:09:46.343850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.343780 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" containerID="cri-o://5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261" gracePeriod=45 Oct 30 06:09:46.666807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.666777 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:09:46.768835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.768796 6171 generic.go:356] "Generic (PLEG): container finished" podID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerID="5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261" exitCode=0 Oct 30 06:09:46.768835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.768833 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" Oct 30 06:09:46.769160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.768879 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" event={"ID":"fd940f31-3819-4f9b-9769-6d9f3e02e31c","Type":"ContainerDied","Data":"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261"} Oct 30 06:09:46.769160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.768925 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq" event={"ID":"fd940f31-3819-4f9b-9769-6d9f3e02e31c","Type":"ContainerDied","Data":"7be2d545264c674f837c1015599428691644a6d73548cba49aa0c32464f0a7a6"} Oct 30 06:09:46.769160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.768946 6171 scope.go:122] "RemoveContainer" containerID="5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261" Oct 30 06:09:46.771158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.771129 6171 generic.go:356] "Generic (PLEG): container finished" podID="2c1ead80-a52a-4bd4-80fc-d815712b7b40" containerID="bd6ab2d58b9e2d5bd7f9af05a8f7128de7b0a25948b31048287d77f81962b712" exitCode=0 Oct 30 06:09:46.771262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.771203 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" event={"ID":"2c1ead80-a52a-4bd4-80fc-d815712b7b40","Type":"ContainerDied","Data":"bd6ab2d58b9e2d5bd7f9af05a8f7128de7b0a25948b31048287d77f81962b712"} Oct 30 06:09:46.778524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.778647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778584 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.778647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778608 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.778647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778637 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778724 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.778824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778783 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:46.778913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778903 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\" (UniqueName: \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\") pod \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\" (UID: \"fd940f31-3819-4f9b-9769-6d9f3e02e31c\") " Oct 30 06:09:46.779021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.778992 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:46.779203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.779025 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:46.779658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.779635 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:46.779760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.779662 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:46.779760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.779680 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:46.781054 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.781019 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:46.781486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.781457 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:46.782221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.782196 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq" pod "fd940f31-3819-4f9b-9769-6d9f3e02e31c" (UID: "fd940f31-3819-4f9b-9769-6d9f3e02e31c"). InnerVolumeSpecName "kube-api-access-ptvxq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:46.783415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.783393 6171 scope.go:122] "RemoveContainer" containerID="5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261" Oct 30 06:09:46.783697 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:46.783676 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261\": container with ID starting with 5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261 not found: ID does not exist" containerID="5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261" Oct 30 06:09:46.783773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.783705 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261"} err="failed to get container status \"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261\": rpc error: code = NotFound desc = could not find container \"5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261\": container with ID starting with 5b39f36e444f518550bcacee627cc4fee910acb1778187e33297035b51ee7261 not found: ID does not exist" Oct 30 06:09:46.881169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.881098 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fd940f31-3819-4f9b-9769-6d9f3e02e31c-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:46.881169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.881124 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptvxq\" (UniqueName: \"kubernetes.io/projected/fd940f31-3819-4f9b-9769-6d9f3e02e31c-kube-api-access-ptvxq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:46.881169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:46.881136 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fd940f31-3819-4f9b-9769-6d9f3e02e31c-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:47.122730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:47.122692 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:09:47.135807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:47.135736 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-c1d09f/test-model-c1d09f-kserve-5598b7b945-chljq"] Oct 30 06:09:47.725398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:47.725336 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" path="/var/lib/kubelet/pods/fd940f31-3819-4f9b-9769-6d9f3e02e31c/volumes" Oct 30 06:09:47.972442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:47.972415 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:48.109341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.109253 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\") pod \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " Oct 30 06:09:48.109530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.109371 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\") pod \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " Oct 30 06:09:48.109608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.109562 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\" (UniqueName: \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\") pod \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\" (UID: \"2c1ead80-a52a-4bd4-80fc-d815712b7b40\") " Oct 30 06:09:48.109792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.109770 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca" pod "2c1ead80-a52a-4bd4-80fc-d815712b7b40" (UID: "2c1ead80-a52a-4bd4-80fc-d815712b7b40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:48.110277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.110213 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:48.111639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.111605 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca" pod "2c1ead80-a52a-4bd4-80fc-d815712b7b40" (UID: "2c1ead80-a52a-4bd4-80fc-d815712b7b40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:48.111717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.111701 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst" pod "2c1ead80-a52a-4bd4-80fc-d815712b7b40" (UID: "2c1ead80-a52a-4bd4-80fc-d815712b7b40"). InnerVolumeSpecName "kube-api-access-8tsst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:48.212417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.212378 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c1ead80-a52a-4bd4-80fc-d815712b7b40-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:48.212417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.212412 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tsst\" (UniqueName: \"kubernetes.io/projected/2c1ead80-a52a-4bd4-80fc-d815712b7b40-kube-api-access-8tsst\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:48.805527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.805488 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" event={"ID":"2c1ead80-a52a-4bd4-80fc-d815712b7b40","Type":"ContainerDied","Data":"9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4"} Oct 30 06:09:48.805527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.805523 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f" Oct 30 06:09:48.805970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:48.805545 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d12e1dab3dcd3ea487903559531f08bf656aed2370efa47d3cffde7a12223d4" Oct 30 06:09:49.679973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.679943 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:09:49.680359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.680341 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:09:49.789242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.789213 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:09:49.790197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790174 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" Oct 30 06:09:49.790197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790193 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" Oct 30 06:09:49.790408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790253 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2c1ead80-a52a-4bd4-80fc-d815712b7b40" containerName="revoke-keys" Oct 30 06:09:49.790408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790258 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c1ead80-a52a-4bd4-80fc-d815712b7b40" containerName="revoke-keys" Oct 30 06:09:49.790491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790408 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c1ead80-a52a-4bd4-80fc-d815712b7b40" containerName="revoke-keys" Oct 30 06:09:49.790491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.790421 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd940f31-3819-4f9b-9769-6d9f3e02e31c" containerName="main" Oct 30 06:09:49.794520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.794500 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.814951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.814921 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:09:49.846552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.846518 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-03d349-openshift-default-dockercfg-2zmxp\"" Oct 30 06:09:49.945224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945030 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjss2\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945191 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945270 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945375 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945450 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945581 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945644 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945692 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:49.945858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:49.945838 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.047703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.047660 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.047904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.047839 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjss2\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.047904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.047869 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048102 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048160 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048210 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048274 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048297 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048326 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048353 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048597 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048350 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048648 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.048833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.048811 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.050800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.050780 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.050914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.050885 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.074334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.074306 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.074519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.074497 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjss2\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\") pod \"e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.114677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.114652 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:50.411642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.411611 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:09:50.446719 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:09:50.446691 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffea1d6_cda5_4edb_a278_268881b3b1bb.slice/crio-02f1f71e2e6fe6f91c66619591897fe1abc70ad364d40d300056d6a9406a8b98 WatchSource:0}: Error finding container 02f1f71e2e6fe6f91c66619591897fe1abc70ad364d40d300056d6a9406a8b98: Status 404 returned error can't find the container with id 02f1f71e2e6fe6f91c66619591897fe1abc70ad364d40d300056d6a9406a8b98 Oct 30 06:09:50.450474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.450433 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:50.450609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.450535 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:50.450609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.450584 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:09:50.451984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.451961 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:09:50.453382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.453360 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:09:50.561172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.560686 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\") pod \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " Oct 30 06:09:50.561172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.560955 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\" (UniqueName: \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\") pod \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " Oct 30 06:09:50.561172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.561066 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\") pod \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\" (UID: \"4e168cfd-a88d-4e77-9ee5-171c88fd1af8\") " Oct 30 06:09:50.561172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.561162 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\" (UniqueName: \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\") pod \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\" (UID: \"8756413e-7f5a-4f30-b9ad-2c1903c4a3da\") " Oct 30 06:09:50.564583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.563556 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume" pod "4e168cfd-a88d-4e77-9ee5-171c88fd1af8" (UID: "4e168cfd-a88d-4e77-9ee5-171c88fd1af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:50.564583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.563983 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume" pod "8756413e-7f5a-4f30-b9ad-2c1903c4a3da" (UID: "8756413e-7f5a-4f30-b9ad-2c1903c4a3da"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:50.575263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.575235 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:09:50.575885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.575518 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerName="maas-api" containerID="cri-o://4325a2de7afd7d03c1147c4b8f21b0758e37a27496570c7920cd703405010f38" gracePeriod=30 Oct 30 06:09:50.576630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.576591 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz" pod "4e168cfd-a88d-4e77-9ee5-171c88fd1af8" (UID: "4e168cfd-a88d-4e77-9ee5-171c88fd1af8"). InnerVolumeSpecName "kube-api-access-tpxqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:50.577064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.577029 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6" pod "8756413e-7f5a-4f30-b9ad-2c1903c4a3da" (UID: "8756413e-7f5a-4f30-b9ad-2c1903c4a3da"). InnerVolumeSpecName "kube-api-access-gn8t6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:50.663574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.663473 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tpxqz\" (UniqueName: \"kubernetes.io/projected/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-kube-api-access-tpxqz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:50.663574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.663509 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e168cfd-a88d-4e77-9ee5-171c88fd1af8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:50.663574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.663524 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn8t6\" (UniqueName: \"kubernetes.io/projected/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-kube-api-access-gn8t6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:50.663574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:50.663538 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8756413e-7f5a-4f30-b9ad-2c1903c4a3da-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.124355 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" event={"ID":"3ffea1d6-cda5-4edb-a278-268881b3b1bb","Type":"ContainerStarted","Data":"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.124431 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" event={"ID":"3ffea1d6-cda5-4edb-a278-268881b3b1bb","Type":"ContainerStarted","Data":"02f1f71e2e6fe6f91c66619591897fe1abc70ad364d40d300056d6a9406a8b98"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.127567 6171 generic.go:356] "Generic (PLEG): container finished" podID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerID="8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d" exitCode=0 Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.127764 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" event={"ID":"4e168cfd-a88d-4e77-9ee5-171c88fd1af8","Type":"ContainerDied","Data":"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.127810 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" event={"ID":"4e168cfd-a88d-4e77-9ee5-171c88fd1af8","Type":"ContainerDied","Data":"efe923c1b7baf0e78c58c14318a4fac389dc0560df2803312164d0bc534d80be"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.127841 6171 scope.go:122] "RemoveContainer" containerID="8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d" Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.127986 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m" Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.140332 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.140626 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.140682 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.143570 6171 generic.go:356] "Generic (PLEG): container finished" podID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerID="6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae" exitCode=0 Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.144081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" event={"ID":"8756413e-7f5a-4f30-b9ad-2c1903c4a3da","Type":"ContainerDied","Data":"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.144139 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" event={"ID":"8756413e-7f5a-4f30-b9ad-2c1903c4a3da","Type":"ContainerDied","Data":"4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257"} Oct 30 06:09:51.149094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.144230 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj" Oct 30 06:09:51.180097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.158850 6171 generic.go:356] "Generic (PLEG): container finished" podID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerID="4325a2de7afd7d03c1147c4b8f21b0758e37a27496570c7920cd703405010f38" exitCode=0 Oct 30 06:09:51.180097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.158943 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" event={"ID":"405e2758-598d-40d8-b3e0-7ac155cbe01f","Type":"ContainerDied","Data":"4325a2de7afd7d03c1147c4b8f21b0758e37a27496570c7920cd703405010f38"} Oct 30 06:09:51.242206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.237226 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podStartSLOduration=2.237207511 podStartE2EDuration="2.237207511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:51.223144267 +0000 UTC m=+2249.663521461" watchObservedRunningTime="2026-10-30 06:09:51.237207511 +0000 UTC m=+2249.677584705" Oct 30 06:09:51.499107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.497309 6171 scope.go:122] "RemoveContainer" containerID="8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d" Oct 30 06:09:51.516815 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:51.516769 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d\": container with ID starting with 8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d not found: ID does not exist" containerID="8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d" Oct 30 06:09:51.516995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.516823 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d"} err="failed to get container status \"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d\": rpc error: code = NotFound desc = could not find container \"8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d\": container with ID starting with 8b141f727be5bc26aec4724d081e8429c7886e81d70a05bd0bfb80cb14815a3d not found: ID does not exist" Oct 30 06:09:51.516995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.516849 6171 scope.go:122] "RemoveContainer" containerID="6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae" Oct 30 06:09:51.550898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.550870 6171 scope.go:122] "RemoveContainer" containerID="6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae" Oct 30 06:09:51.551406 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:51.551378 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae\": container with ID starting with 6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae not found: ID does not exist" containerID="6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae" Oct 30 06:09:51.551523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.551419 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae"} err="failed to get container status \"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae\": rpc error: code = NotFound desc = could not find container \"6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae\": container with ID starting with 6fc5076b5f266d5d616bfb2a0de3b56876a4ca21c91866f6da170928a47856ae not found: ID does not exist" Oct 30 06:09:51.552084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.552011 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88"] Oct 30 06:09:51.553247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.553153 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:09:51.560034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.560010 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c54211-84h88"] Oct 30 06:09:51.568993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.568961 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:09:51.577796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.577769 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-c54211-6bf57c4c67-2qn7m"] Oct 30 06:09:51.590895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.590872 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:09:51.591013 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:51.590927 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756413e_7f5a_4f30_b9ad_2c1903c4a3da.slice/crio-4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756413e_7f5a_4f30_b9ad_2c1903c4a3da.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:09:51.591013 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:51.590965 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8756413e_7f5a_4f30_b9ad_2c1903c4a3da.slice/crio-4198b91896c77a5164d772a30cdc5e27fc3441becf437d2b8e48f032f39e0257\": RecentStats: unable to find data in memory cache]" Oct 30 06:09:51.598730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.598709 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-c54211-7c65667ccf-bcmkj"] Oct 30 06:09:51.619685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.619654 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\" (UniqueName: \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\") pod \"405e2758-598d-40d8-b3e0-7ac155cbe01f\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " Oct 30 06:09:51.619851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.619751 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\") pod \"405e2758-598d-40d8-b3e0-7ac155cbe01f\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " Oct 30 06:09:51.619851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.619779 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\") pod \"405e2758-598d-40d8-b3e0-7ac155cbe01f\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " Oct 30 06:09:51.619963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.619920 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\") pod \"405e2758-598d-40d8-b3e0-7ac155cbe01f\" (UID: \"405e2758-598d-40d8-b3e0-7ac155cbe01f\") " Oct 30 06:09:51.620131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.620102 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca" pod "405e2758-598d-40d8-b3e0-7ac155cbe01f" (UID: "405e2758-598d-40d8-b3e0-7ac155cbe01f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:51.620743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.620674 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/405e2758-598d-40d8-b3e0-7ac155cbe01f-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:51.623457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.623427 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr" pod "405e2758-598d-40d8-b3e0-7ac155cbe01f" (UID: "405e2758-598d-40d8-b3e0-7ac155cbe01f"). InnerVolumeSpecName "kube-api-access-p2xmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:51.623552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.623497 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs" pod "405e2758-598d-40d8-b3e0-7ac155cbe01f" (UID: "405e2758-598d-40d8-b3e0-7ac155cbe01f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:51.623552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.623521 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls" pod "405e2758-598d-40d8-b3e0-7ac155cbe01f" (UID: "405e2758-598d-40d8-b3e0-7ac155cbe01f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.719545 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" path="/var/lib/kubelet/pods/4e168cfd-a88d-4e77-9ee5-171c88fd1af8/volumes" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.720388 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54e72a7e-08c2-4deb-aad9-de6acb63cd9e" path="/var/lib/kubelet/pods/54e72a7e-08c2-4deb-aad9-de6acb63cd9e/volumes" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.721009 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" path="/var/lib/kubelet/pods/8756413e-7f5a-4f30-b9ad-2c1903c4a3da/volumes" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.723083 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2xmr\" (UniqueName: \"kubernetes.io/projected/405e2758-598d-40d8-b3e0-7ac155cbe01f-kube-api-access-p2xmr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.723109 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:51.723118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:51.723125 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/405e2758-598d-40d8-b3e0-7ac155cbe01f-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:52.115683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.115583 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:52.115683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.115649 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:52.194806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.194773 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" Oct 30 06:09:52.196178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.196152 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" event={"ID":"405e2758-598d-40d8-b3e0-7ac155cbe01f","Type":"ContainerDied","Data":"47564442e58cfb5392cde4e88d63156a846e4af5448554c35744a127b21df649"} Oct 30 06:09:52.196360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.196346 6171 scope.go:122] "RemoveContainer" containerID="4325a2de7afd7d03c1147c4b8f21b0758e37a27496570c7920cd703405010f38" Oct 30 06:09:52.328106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.324114 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:09:52.365449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:52.365409 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x"] Oct 30 06:09:53.116022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:53.115986 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:53.116226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:53.116055 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:53.676898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:53.676857 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" path="/var/lib/kubelet/pods/405e2758-598d-40d8-b3e0-7ac155cbe01f/volumes" Oct 30 06:09:54.115533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:54.115496 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:54.115723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:54.115560 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:54.405488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:54.405379 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-c54211-67b657bcc8-9b75x" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.252:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:09:55.115606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:55.115573 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:55.116012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:55.115627 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:56.115285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:56.115245 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:56.115468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:56.115304 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:56.409887 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:56.409793 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:09:57.115623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:57.115583 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" start-of-body= Oct 30 06:09:57.115814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:57.115657 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.40:15021/healthz/ready\": dial tcp 10.133.1.40:15021: connect: connection refused" Oct 30 06:09:58.296392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:58.296355 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:58.298982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:58.298943 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:58.347591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:58.335245 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:09:58.943488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:58.943444 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" containerID="cri-o://3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0" gracePeriod=45 Oct 30 06:09:59.300113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.300082 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:09:59.374171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.374134 6171 generic.go:356] "Generic (PLEG): container finished" podID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerID="3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0" exitCode=0 Oct 30 06:09:59.374349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.374187 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" Oct 30 06:09:59.374349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.374175 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" event={"ID":"b32ad1e0-cd7c-44fb-b35f-0f4785910771","Type":"ContainerDied","Data":"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0"} Oct 30 06:09:59.374349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.374314 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" event={"ID":"b32ad1e0-cd7c-44fb-b35f-0f4785910771","Type":"ContainerDied","Data":"69850e3c3a5417ed86da0c2f9bbc9803f3e41c8192020d16fdb02d7f0b908040"} Oct 30 06:09:59.374349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.374342 6171 scope.go:122] "RemoveContainer" containerID="3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0" Oct 30 06:09:59.386503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.386480 6171 scope.go:122] "RemoveContainer" containerID="3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0" Oct 30 06:09:59.386859 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:09:59.386831 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0\": container with ID starting with 3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0 not found: ID does not exist" containerID="3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0" Oct 30 06:09:59.386969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.386861 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0"} err="failed to get container status \"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0\": rpc error: code = NotFound desc = could not find container \"3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0\": container with ID starting with 3bf6e727ca06b661ef3cae58581c0dd39403bb2e10753f3995d81e0f6fce71d0 not found: ID does not exist" Oct 30 06:09:59.436486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.436451 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.436930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.436906 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.437207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.437188 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\" (UniqueName: \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.437464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.437446 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.437625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.437609 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\" (UniqueName: \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.437761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.437745 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\") pod \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\" (UID: \"b32ad1e0-cd7c-44fb-b35f-0f4785910771\") " Oct 30 06:09:59.439778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.439682 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:59.440153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.440107 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:59.440928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.440907 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:59.457902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.457698 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "kube-api-access-wt8bp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:59.469495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.468322 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:09:59.469689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.469429 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:59.469767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.469541 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm" pod "b32ad1e0-cd7c-44fb-b35f-0f4785910771" (UID: "b32ad1e0-cd7c-44fb-b35f-0f4785910771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:59.469915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.469832 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" containerID="cri-o://85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe" gracePeriod=30 Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540605 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540633 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540642 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540652 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540661 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt8bp\" (UniqueName: \"kubernetes.io/projected/b32ad1e0-cd7c-44fb-b35f-0f4785910771-kube-api-access-wt8bp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.540672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.540671 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b32ad1e0-cd7c-44fb-b35f-0f4785910771-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:09:59.865221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.865122 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:09:59.874428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:09:59.874398 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74"] Oct 30 06:10:00.150225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.150134 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-7df9fa/test-model-7df9fa-kserve-74c8d484df-m8f74" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" probeResult="failure" output="Get \"https://10.133.1.13:8000/ready\": context deadline exceeded" Oct 30 06:10:00.864109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.864068 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:10:00.865305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865255 6171 cpu_manager.go:414] "containerMap: removing container" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerName="maas-api" Oct 30 06:10:00.865305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865281 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerName="maas-api" Oct 30 06:10:00.865305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865297 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" Oct 30 06:10:00.865305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865302 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" Oct 30 06:10:00.865305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865313 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865319 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865396 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865405 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865609 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" containerName="main" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865630 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="405e2758-598d-40d8-b3e0-7ac155cbe01f" containerName="maas-api" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865640 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8756413e-7f5a-4f30-b9ad-2c1903c4a3da" containerName="payload-processing" Oct 30 06:10:00.865717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.865649 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e168cfd-a88d-4e77-9ee5-171c88fd1af8" containerName="payload-pre-processing" Oct 30 06:10:00.869947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.869928 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:00.899709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.899669 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:10:00.904969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:00.904939 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-24db22-openshift-default-dockercfg-75mbl\"" Oct 30 06:10:01.043071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.004726 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077257 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077583 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svkjh\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077657 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077761 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077792 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077869 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077913 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.088073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.077985 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.186119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.185662 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.186529 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.186808 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svkjh\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.186861 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.186954 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.186982 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.188837 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.190067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.189374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.205769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.205698 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.206107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.206084 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.206355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.206336 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.207014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.206863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.207606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.207212 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.207900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.207576 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.246081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.245277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.255910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.255870 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svkjh\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.261087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.259753 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.261934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.261907 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\") pod \"e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.486312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.486283 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:01.675961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.675930 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b32ad1e0-cd7c-44fb-b35f-0f4785910771" path="/var/lib/kubelet/pods/b32ad1e0-cd7c-44fb-b35f-0f4785910771/volumes" Oct 30 06:10:01.826115 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:01.826085 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39d80a0_0fb5_49f5_a77a_b5e54e700e62.slice/crio-779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75 WatchSource:0}: Error finding container 779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75: Status 404 returned error can't find the container with id 779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75 Oct 30 06:10:01.828089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.828058 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:10:01.829086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.829032 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:01.829235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.829131 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:01.829504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:01.829177 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:02.422742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:02.422694 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerStarted","Data":"939405f21115328c1b3a1001de04c79b481a9a8c1474879382f2e5606dd07805"} Oct 30 06:10:02.423229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:02.422770 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerStarted","Data":"779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75"} Oct 30 06:10:02.474194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:02.474136 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podStartSLOduration=2.474117981 podStartE2EDuration="2.474117981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:02.471093122 +0000 UTC m=+2260.911470312" watchObservedRunningTime="2026-10-30 06:10:02.474117981 +0000 UTC m=+2260.914495173" Oct 30 06:10:02.487492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:02.487454 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:02.494812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:02.494782 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:03.470294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:03.470258 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:03.471436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:03.471414 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:10:04.779483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.779440 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:10:04.856318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856262 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856353 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856576 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856677 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856669 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.856823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856742 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856783 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.856823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856813 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.857246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.856923 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.857246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.857102 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\") pod \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\" (UID: \"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50\") " Oct 30 06:10:04.858145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.858117 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.858349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.858333 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.858933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.858877 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.859131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.858975 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:04.859228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.859122 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.859758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.859731 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:04.859867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.859767 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:04.861250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.861222 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "kube-api-access-6qfrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:04.862550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.862496 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy" pod "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" (UID: "a91bdae9-a4d3-4b3d-aa6f-2720e391dc50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.960956 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.960996 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.961010 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.961023 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.961037 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qfrv\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-kube-api-access-6qfrv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.961111 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:04.961368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:04.961121 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:05.492092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.492023 6171 generic.go:356] "Generic (PLEG): container finished" podID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerID="85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe" exitCode=0 Oct 30 06:10:05.492092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.492081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerDied","Data":"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe"} Oct 30 06:10:05.492092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.492098 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" Oct 30 06:10:05.492461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.492146 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t" event={"ID":"a91bdae9-a4d3-4b3d-aa6f-2720e391dc50","Type":"ContainerDied","Data":"7fda8c1ffeb98f16da5bd2637206b9dc60d6613b999382343ad6f18b498a8608"} Oct 30 06:10:05.492461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.492173 6171 scope.go:122] "RemoveContainer" containerID="85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe" Oct 30 06:10:05.506978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.506956 6171 scope.go:122] "RemoveContainer" containerID="74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5" Oct 30 06:10:05.520582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.520554 6171 scope.go:122] "RemoveContainer" containerID="85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe" Oct 30 06:10:05.520936 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:05.520917 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe\": container with ID starting with 85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe not found: ID does not exist" containerID="85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe" Oct 30 06:10:05.521030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.520944 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe"} err="failed to get container status \"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe\": rpc error: code = NotFound desc = could not find container \"85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe\": container with ID starting with 85d1152740d5c6fa6efcd0c80e630ed6088135521a7fe8d417ae21c7581f90fe not found: ID does not exist" Oct 30 06:10:05.521030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.520962 6171 scope.go:122] "RemoveContainer" containerID="74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5" Oct 30 06:10:05.521339 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:05.521312 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5\": container with ID starting with 74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5 not found: ID does not exist" containerID="74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5" Oct 30 06:10:05.521408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.521346 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5"} err="failed to get container status \"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5\": rpc error: code = NotFound desc = could not find container \"74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5\": container with ID starting with 74cdb771abe3f6b2e02006403b041d0cd3f5a39ac9517eb6b31d7ac6d85032c5 not found: ID does not exist" Oct 30 06:10:05.569356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.569217 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:10:05.575732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.575699 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-c54211-openshift-default-767566b477-nvw4t"] Oct 30 06:10:05.676304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:05.676271 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" path="/var/lib/kubelet/pods/a91bdae9-a4d3-4b3d-aa6f-2720e391dc50/volumes" Oct 30 06:10:06.609306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.609269 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:10:06.610216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.610199 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.610299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.610218 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.610385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.610375 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.610435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.610391 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.616225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.616192 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:06.638557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.638524 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:10:06.645444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.645411 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-24db22-dockercfg-qzfpx\"" Oct 30 06:10:06.645444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.645408 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-24db22\"" Oct 30 06:10:06.688592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.688553 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:10:06.690138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.690104 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.690138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.690133 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a91bdae9-a4d3-4b3d-aa6f-2720e391dc50" containerName="istio-proxy" Oct 30 06:10:06.696003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.695973 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:06.712524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.712483 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g728q\" (UniqueName: \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:06.712733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.712593 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:06.836070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.834186 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:06.836070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.834286 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:06.836070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.834562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnb2n\" (UniqueName: \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:06.836070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.834876 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g728q\" (UniqueName: \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:07.066657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:06.998322 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:07.070868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.004575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:07.071708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.071468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnb2n\" (UniqueName: \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:07.093138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.005630 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:07.093138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.086619 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:10:07.144073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.137510 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnb2n\" (UniqueName: \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\") pod \"payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:07.161870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.160468 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g728q\" (UniqueName: \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\") pod \"payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:07.234335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.234295 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:07.339508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.339467 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:07.582585 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:07.582518 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93a08c43_2493_47c8_9501_ed90af6f9972.slice/crio-91b00b3919ccb61adff36070d2a675533c47c7e4c70061c7e07f7356d57e1e8a WatchSource:0}: Error finding container 91b00b3919ccb61adff36070d2a675533c47c7e4c70061c7e07f7356d57e1e8a: Status 404 returned error can't find the container with id 91b00b3919ccb61adff36070d2a675533c47c7e4c70061c7e07f7356d57e1e8a Oct 30 06:10:07.583533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.583508 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:10:07.679786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:07.679757 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:10:08.563068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.562759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" event={"ID":"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6","Type":"ContainerStarted","Data":"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581"} Oct 30 06:10:08.563068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.562834 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" event={"ID":"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6","Type":"ContainerStarted","Data":"1f7ea6297e7272de346229af62aee1f0d264685de69818c7b6341a28267f9b80"} Oct 30 06:10:08.568068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.564182 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:08.576221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.575112 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" event={"ID":"93a08c43-2493-47c8-9501-ed90af6f9972","Type":"ContainerStarted","Data":"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246"} Oct 30 06:10:08.576221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.575174 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" event={"ID":"93a08c43-2493-47c8-9501-ed90af6f9972","Type":"ContainerStarted","Data":"91b00b3919ccb61adff36070d2a675533c47c7e4c70061c7e07f7356d57e1e8a"} Oct 30 06:10:08.576221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.575578 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:08.611085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.610998 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" podStartSLOduration=2.610979558 podStartE2EDuration="2.610979558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:08.607946056 +0000 UTC m=+2267.048323249" watchObservedRunningTime="2026-10-30 06:10:08.610979558 +0000 UTC m=+2267.051356754" Oct 30 06:10:08.655897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:08.655841 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" podStartSLOduration=2.655825507 podStartE2EDuration="2.655825507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:08.651669499 +0000 UTC m=+2267.092046693" watchObservedRunningTime="2026-10-30 06:10:08.655825507 +0000 UTC m=+2267.096202700" Oct 30 06:10:10.588978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.588941 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 06:10:10.589484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.589212 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" containerID="cri-o://8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331" gracePeriod=30 Oct 30 06:10:10.633534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.633476 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 06:10:10.633871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.633841 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" containerID="cri-o://fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98" gracePeriod=30 Oct 30 06:10:10.728738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.724980 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:10:10.950193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:10.950101 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv"] Oct 30 06:10:11.023913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.023878 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.026174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.025530 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv"] Oct 30 06:10:11.047715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.047686 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:11.047897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.047828 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:10:11.125057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.125006 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.125243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.125071 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.125243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.125188 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhjbk\" (UniqueName: \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.227431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.227397 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.227616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.227452 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.227616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.227509 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhjbk\" (UniqueName: \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.228079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.228023 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.229990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.229972 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.252640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.252613 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhjbk\" (UniqueName: \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\") pod \"maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.355353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.355318 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:11.669151 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:11.669095 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod816d1cb9_7611_42b2_b829_ee145eb2e1af.slice/crio-15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3 WatchSource:0}: Error finding container 15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3: Status 404 returned error can't find the container with id 15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3 Oct 30 06:10:11.679549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.679522 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv"] Oct 30 06:10:11.959931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.959849 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:10:11.966658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.966631 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:11.987563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.987421 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-d1bfce-serving-cert\"" Oct 30 06:10:11.990561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:11.990528 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:10:12.041406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.041342 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.041574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.041517 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k7sw\" (UniqueName: \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.041691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.041678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.041825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.041803 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.143559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.143525 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.143844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.143656 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.143844 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:12.143689 6171 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-serving-cert: secret "maas-api-e2e-shared-a-w5-d1bfce-serving-cert" not found Oct 30 06:10:12.143844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.143723 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.143844 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:12.143762 6171 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls podName:d6594fe0-7c2f-4344-9167-0d507b0232ce nodeName:}" failed. No retries permitted until 2026-10-30 06:10:12.643738651 +0000 UTC m=+2271.084115838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls") pod "maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" (UID: "d6594fe0-7c2f-4344-9167-0d507b0232ce") : secret "maas-api-e2e-shared-a-w5-d1bfce-serving-cert" not found Oct 30 06:10:12.144177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.143936 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k7sw\" (UniqueName: \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.144518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.144494 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.146669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.146649 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.170845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.170817 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k7sw\" (UniqueName: \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.625266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.625224 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" event={"ID":"816d1cb9-7611-42b2-b829-ee145eb2e1af","Type":"ContainerStarted","Data":"5939fde15d43bb91f502c0ff16eef8d66806fc43f0deba0ac1fbfb38a04769b1"} Oct 30 06:10:12.625499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.625350 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" event={"ID":"816d1cb9-7611-42b2-b829-ee145eb2e1af","Type":"ContainerStarted","Data":"15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3"} Oct 30 06:10:12.652039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.652005 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.654572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.654550 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:12.679377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.679326 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" podStartSLOduration=2.6793142899999998 podStartE2EDuration="2.67931429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:12.674913086 +0000 UTC m=+2271.115290279" watchObservedRunningTime="2026-10-30 06:10:12.67931429 +0000 UTC m=+2271.119691501" Oct 30 06:10:12.889255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:12.889180 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:13.253842 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:13.253811 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6594fe0_7c2f_4344_9167_0d507b0232ce.slice/crio-b6bea80b939c5d031dfa4140373f93393bcf2b0b7b9c0f2677ffb4f88530455a WatchSource:0}: Error finding container b6bea80b939c5d031dfa4140373f93393bcf2b0b7b9c0f2677ffb4f88530455a: Status 404 returned error can't find the container with id b6bea80b939c5d031dfa4140373f93393bcf2b0b7b9c0f2677ffb4f88530455a Oct 30 06:10:13.255319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:13.255291 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:10:13.255842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:13.255821 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:10:13.643979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:13.643892 6171 generic.go:356] "Generic (PLEG): container finished" podID="816d1cb9-7611-42b2-b829-ee145eb2e1af" containerID="5939fde15d43bb91f502c0ff16eef8d66806fc43f0deba0ac1fbfb38a04769b1" exitCode=0 Oct 30 06:10:13.644202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:13.643985 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" event={"ID":"816d1cb9-7611-42b2-b829-ee145eb2e1af","Type":"ContainerDied","Data":"5939fde15d43bb91f502c0ff16eef8d66806fc43f0deba0ac1fbfb38a04769b1"} Oct 30 06:10:13.645458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:13.645434 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" event={"ID":"d6594fe0-7c2f-4344-9167-0d507b0232ce","Type":"ContainerStarted","Data":"b6bea80b939c5d031dfa4140373f93393bcf2b0b7b9c0f2677ffb4f88530455a"} Oct 30 06:10:14.656658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:14.656615 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" event={"ID":"d6594fe0-7c2f-4344-9167-0d507b0232ce","Type":"ContainerStarted","Data":"4633f60de251c6fcb5c5dec4342b7d030b096aadbc8e03430bc7be2498085e6e"} Oct 30 06:10:14.657174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:14.656709 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:14.710023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:14.709966 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" podStartSLOduration=3.306579704 podStartE2EDuration="3.709951602s" totalImagesPullingTime="403.371898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:14.703702797 +0000 UTC m=+2273.144079990" watchObservedRunningTime="2026-10-30 06:10:14.709951602 +0000 UTC m=+2273.150328795" Oct 30 06:10:14.866964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:14.866936 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:15.052844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.052805 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\") pod \"816d1cb9-7611-42b2-b829-ee145eb2e1af\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " Oct 30 06:10:15.053101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.052991 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\") pod \"816d1cb9-7611-42b2-b829-ee145eb2e1af\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " Oct 30 06:10:15.053101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.053086 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\" (UniqueName: \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\") pod \"816d1cb9-7611-42b2-b829-ee145eb2e1af\" (UID: \"816d1cb9-7611-42b2-b829-ee145eb2e1af\") " Oct 30 06:10:15.053612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.053577 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca" pod "816d1cb9-7611-42b2-b829-ee145eb2e1af" (UID: "816d1cb9-7611-42b2-b829-ee145eb2e1af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:15.054215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.054190 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:15.069232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.069194 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca" pod "816d1cb9-7611-42b2-b829-ee145eb2e1af" (UID: "816d1cb9-7611-42b2-b829-ee145eb2e1af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:15.069398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.069334 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk" pod "816d1cb9-7611-42b2-b829-ee145eb2e1af" (UID: "816d1cb9-7611-42b2-b829-ee145eb2e1af"). InnerVolumeSpecName "kube-api-access-zhjbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:15.156355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.156314 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/816d1cb9-7611-42b2-b829-ee145eb2e1af-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:15.156355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.156356 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhjbk\" (UniqueName: \"kubernetes.io/projected/816d1cb9-7611-42b2-b829-ee145eb2e1af-kube-api-access-zhjbk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:15.308135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.303805 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz"] Oct 30 06:10:15.308135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.305341 6171 cpu_manager.go:414] "containerMap: removing container" podUID="816d1cb9-7611-42b2-b829-ee145eb2e1af" containerName="revoke-keys" Oct 30 06:10:15.308135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.305365 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="816d1cb9-7611-42b2-b829-ee145eb2e1af" containerName="revoke-keys" Oct 30 06:10:15.308135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.305619 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="816d1cb9-7611-42b2-b829-ee145eb2e1af" containerName="revoke-keys" Oct 30 06:10:15.338451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.338409 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.389725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.389687 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz"] Oct 30 06:10:15.465756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.465717 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.465972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.465926 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.466234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.466205 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzzqh\" (UniqueName: \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.510425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.510385 6171 patch_prober.go:29] interesting pod/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:15.510626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.510450 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:15.568535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.568431 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzzqh\" (UniqueName: \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.568723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.568569 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.568723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.568702 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.569277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.569253 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.571397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.571376 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.594542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.594505 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzzqh\" (UniqueName: \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\") pod \"maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.669848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.669388 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" Oct 30 06:10:15.676439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.676409 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv" event={"ID":"816d1cb9-7611-42b2-b829-ee145eb2e1af","Type":"ContainerDied","Data":"15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3"} Oct 30 06:10:15.676601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.676457 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="15285aaee83ce098afab15d04f8b0462066d2637da1ae0f265b645f6235acdf3" Oct 30 06:10:15.691836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.691801 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:15.959589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.959553 6171 patch_prober.go:29] interesting pod/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" start-of-body= Oct 30 06:10:15.959903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:15.959868 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.81:15021/healthz/ready\": dial tcp 10.133.0.81:15021: connect: connection refused" Oct 30 06:10:16.144705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.144678 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 06:10:16.148828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.148804 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 06:10:16.219803 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:16.219768 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a70b3d_16f8_4765_acf8_35bb6d3a56f7.slice/crio-50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e WatchSource:0}: Error finding container 50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e: Status 404 returned error can't find the container with id 50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e Oct 30 06:10:16.222412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.222387 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz"] Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288160 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288256 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288372 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288414 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288459 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288494 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288648 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288697 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288722 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288753 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288818 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288931 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288963 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.288994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.289106 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.289210 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\") pod \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\" (UID: \"33f0043c-6834-4bf4-88a3-fdd1662b51ff\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.289310 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\") pod \"79ff11b1-4490-4b67-b1fe-2b319f67f239\" (UID: \"79ff11b1-4490-4b67-b1fe-2b319f67f239\") " Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.293623 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.294197 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.294454 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.294673 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.294618 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.295224 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.295495 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.296299 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.297739 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.298118 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.300684 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.301806 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.301834 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.301914 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.302153 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "kube-api-access-vxqg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.302825 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token" pod "33f0043c-6834-4bf4-88a3-fdd1662b51ff" (UID: "33f0043c-6834-4bf4-88a3-fdd1662b51ff"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:16.303712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.303668 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "kube-api-access-jl6vd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:16.307365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.307330 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy" pod "79ff11b1-4490-4b67-b1fe-2b319f67f239" (UID: "79ff11b1-4490-4b67-b1fe-2b319f67f239"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392412 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392443 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jl6vd\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-kube-api-access-jl6vd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392453 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392464 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392472 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392479 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392487 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392496 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxqg6\" (UniqueName: \"kubernetes.io/projected/33f0043c-6834-4bf4-88a3-fdd1662b51ff-kube-api-access-vxqg6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392504 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392512 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392521 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392529 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33f0043c-6834-4bf4-88a3-fdd1662b51ff-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392536 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79ff11b1-4490-4b67-b1fe-2b319f67f239-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392544 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392551 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392559 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79ff11b1-4490-4b67-b1fe-2b319f67f239-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392567 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33f0043c-6834-4bf4-88a3-fdd1662b51ff-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.392957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.392575 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79ff11b1-4490-4b67-b1fe-2b319f67f239-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:16.682716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.682628 6171 generic.go:356] "Generic (PLEG): container finished" podID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerID="fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98" exitCode=0 Oct 30 06:10:16.682716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.682695 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" Oct 30 06:10:16.683317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.682720 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" event={"ID":"79ff11b1-4490-4b67-b1fe-2b319f67f239","Type":"ContainerDied","Data":"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98"} Oct 30 06:10:16.683317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.682770 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw" event={"ID":"79ff11b1-4490-4b67-b1fe-2b319f67f239","Type":"ContainerDied","Data":"4dd2f3f870469b9bf2d3a5ce00d30d1ae1d55ec2c871f24297b54cb40de0e70e"} Oct 30 06:10:16.683317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.682798 6171 scope.go:122] "RemoveContainer" containerID="fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98" Oct 30 06:10:16.685101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.685034 6171 generic.go:356] "Generic (PLEG): container finished" podID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerID="8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331" exitCode=0 Oct 30 06:10:16.685225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.685149 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" Oct 30 06:10:16.685296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.685074 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" event={"ID":"33f0043c-6834-4bf4-88a3-fdd1662b51ff","Type":"ContainerDied","Data":"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331"} Oct 30 06:10:16.685522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.685330 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z" event={"ID":"33f0043c-6834-4bf4-88a3-fdd1662b51ff","Type":"ContainerDied","Data":"20c525c94c24614763da09164da3d66f0f04e92435222a03f72ce7c17a678b3a"} Oct 30 06:10:16.691508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.691481 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" event={"ID":"96a70b3d-16f8-4765-acf8-35bb6d3a56f7","Type":"ContainerStarted","Data":"50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e"} Oct 30 06:10:16.707231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.706912 6171 scope.go:122] "RemoveContainer" containerID="fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98" Oct 30 06:10:16.707671 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:16.707648 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98\": container with ID starting with fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98 not found: ID does not exist" containerID="fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98" Oct 30 06:10:16.707772 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.707683 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98"} err="failed to get container status \"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98\": rpc error: code = NotFound desc = could not find container \"fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98\": container with ID starting with fc05e6c45e58c8c74608ae3fea90196ab8a2718efd1f21463b751f62382e3e98 not found: ID does not exist" Oct 30 06:10:16.707772 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.707704 6171 scope.go:122] "RemoveContainer" containerID="8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331" Oct 30 06:10:16.719449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.719425 6171 scope.go:122] "RemoveContainer" containerID="8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331" Oct 30 06:10:16.719786 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:16.719765 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331\": container with ID starting with 8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331 not found: ID does not exist" containerID="8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331" Oct 30 06:10:16.719864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.719803 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331"} err="failed to get container status \"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331\": rpc error: code = NotFound desc = could not find container \"8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331\": container with ID starting with 8b326763ddbfa76b3e0d1818053f35bf01f79e598e059f33d30ffbc49a38c331 not found: ID does not exist" Oct 30 06:10:16.753529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.753497 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 06:10:16.758843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.758811 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-r77zw"] Oct 30 06:10:16.795985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.795948 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 06:10:16.801158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:16.801133 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-128436-openshift-default-fd9c97c99-x2s4z"] Oct 30 06:10:17.675260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:17.675171 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" path="/var/lib/kubelet/pods/33f0043c-6834-4bf4-88a3-fdd1662b51ff/volumes" Oct 30 06:10:17.675876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:17.675850 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" path="/var/lib/kubelet/pods/79ff11b1-4490-4b67-b1fe-2b319f67f239/volumes" Oct 30 06:10:17.705087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:17.705020 6171 generic.go:356] "Generic (PLEG): container finished" podID="96a70b3d-16f8-4765-acf8-35bb6d3a56f7" containerID="2ee0ee2bd346637a2d130aadd6e997145223f3267c5f54eb365a6701647237eb" exitCode=0 Oct 30 06:10:17.705510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:17.705111 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" event={"ID":"96a70b3d-16f8-4765-acf8-35bb6d3a56f7","Type":"ContainerDied","Data":"2ee0ee2bd346637a2d130aadd6e997145223f3267c5f54eb365a6701647237eb"} Oct 30 06:10:18.890282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:18.890257 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:19.048069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.040324 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\" (UniqueName: \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\") pod \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " Oct 30 06:10:19.048069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.040433 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\") pod \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " Oct 30 06:10:19.048069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.040494 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\") pod \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\" (UID: \"96a70b3d-16f8-4765-acf8-35bb6d3a56f7\") " Oct 30 06:10:19.077652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.071684 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca" pod "96a70b3d-16f8-4765-acf8-35bb6d3a56f7" (UID: "96a70b3d-16f8-4765-acf8-35bb6d3a56f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:19.077652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.072353 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:19.105627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.105520 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca" pod "96a70b3d-16f8-4765-acf8-35bb6d3a56f7" (UID: "96a70b3d-16f8-4765-acf8-35bb6d3a56f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:19.105627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.105571 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh" pod "96a70b3d-16f8-4765-acf8-35bb6d3a56f7" (UID: "96a70b3d-16f8-4765-acf8-35bb6d3a56f7"). InnerVolumeSpecName "kube-api-access-vzzqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:19.174033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.174006 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzzqh\" (UniqueName: \"kubernetes.io/projected/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-kube-api-access-vzzqh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:19.174033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.174034 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/96a70b3d-16f8-4765-acf8-35bb6d3a56f7-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:19.589383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.589356 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:10:19.726133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.726103 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" Oct 30 06:10:19.726339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.726199 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz" event={"ID":"96a70b3d-16f8-4765-acf8-35bb6d3a56f7","Type":"ContainerDied","Data":"50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e"} Oct 30 06:10:19.726339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.726264 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50a827f0f39e47c96d60f6c0bef39de1e84043111a8b6ab55aa8a87b3032908e" Oct 30 06:10:19.978565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:19.978529 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:10:20.600682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:20.600656 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:10:20.677968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:20.677942 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:10:21.306280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:21.306238 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:10:24.025501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:24.025455 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 06:10:24.026022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:24.025733 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" containerID="cri-o://f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb" gracePeriod=30 Oct 30 06:10:24.569730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:24.569692 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv"] Oct 30 06:10:24.587954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:24.587859 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-c1d09f-z6mtv"] Oct 30 06:10:25.676664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:25.676625 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="816d1cb9-7611-42b2-b829-ee145eb2e1af" path="/var/lib/kubelet/pods/816d1cb9-7611-42b2-b829-ee145eb2e1af/volumes" Oct 30 06:10:25.749863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:25.749364 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" containerID="cri-o://6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b" gracePeriod=45 Oct 30 06:10:26.144362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.144321 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:10:26.157550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157521 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.157696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157580 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.157696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157617 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\" (UniqueName: \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.157696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157665 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.157897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157762 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:26.157897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157812 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.157897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157848 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\") pod \"ffc9eee7-3498-464d-bff5-82b4d48a1360\" (UID: \"ffc9eee7-3498-464d-bff5-82b4d48a1360\") " Oct 30 06:10:26.158087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.157971 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:26.158087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.158038 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:26.158638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.158613 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.158638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.158639 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.158778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.158647 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.160321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.160299 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:26.160648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.160621 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:26.160741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.160701 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9" pod "ffc9eee7-3498-464d-bff5-82b4d48a1360" (UID: "ffc9eee7-3498-464d-bff5-82b4d48a1360"). InnerVolumeSpecName "kube-api-access-f6xt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:26.354121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.353022 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ffc9eee7-3498-464d-bff5-82b4d48a1360-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.354121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.353080 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6xt9\" (UniqueName: \"kubernetes.io/projected/ffc9eee7-3498-464d-bff5-82b4d48a1360-kube-api-access-f6xt9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.354121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.353094 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ffc9eee7-3498-464d-bff5-82b4d48a1360-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:26.392084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.389870 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" containerID="cri-o://17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946" gracePeriod=25 Oct 30 06:10:26.543544 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:26.543508 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:10:26.870053 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.870014 6171 generic.go:356] "Generic (PLEG): container finished" podID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerID="6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b" exitCode=0 Oct 30 06:10:26.870566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.870082 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" event={"ID":"ffc9eee7-3498-464d-bff5-82b4d48a1360","Type":"ContainerDied","Data":"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b"} Oct 30 06:10:26.870566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.870140 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" event={"ID":"ffc9eee7-3498-464d-bff5-82b4d48a1360","Type":"ContainerDied","Data":"f4612e9e903d37fcc07fd0cc3f2f1527df7ecf847142c8b883d63fbab60bf847"} Oct 30 06:10:26.870566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.870158 6171 scope.go:122] "RemoveContainer" containerID="6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b" Oct 30 06:10:26.870566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.870094 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p" Oct 30 06:10:26.885871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.885805 6171 scope.go:122] "RemoveContainer" containerID="6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b" Oct 30 06:10:26.887076 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:26.887015 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b\": container with ID starting with 6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b not found: ID does not exist" containerID="6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b" Oct 30 06:10:26.887180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.887075 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b"} err="failed to get container status \"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b\": rpc error: code = NotFound desc = could not find container \"6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b\": container with ID starting with 6c996d6b7f8c1158269dbb960f47fa42e925e58977d991b016af5a69738e9a8b not found: ID does not exist" Oct 30 06:10:26.930624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.930589 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:10:26.935514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:26.935491 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-62316c/test-model-62316c-kserve-6bd459fb77-t4d7p"] Oct 30 06:10:27.675458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:27.675425 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" path="/var/lib/kubelet/pods/ffc9eee7-3498-464d-bff5-82b4d48a1360/volumes" Oct 30 06:10:27.812780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:27.812744 6171 patch_prober.go:29] interesting pod/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:27.812954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:27.812804 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:27.875158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:27.875130 6171 scope.go:122] "RemoveContainer" containerID="e1fe214be631f1d8fdd02a072c6a184fc2c0f012ee371b34b7d4c38dc1813956" Oct 30 06:10:28.470279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:28.470241 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:29.748058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.748014 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 06:10:29.813663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813633 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.813810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813716 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.813810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813782 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.813905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813863 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.813950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813932 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.813950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813930 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:29.814035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.813937 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:29.814035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814023 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.814171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814094 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.814171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814139 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.814275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\") pod \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\" (UID: \"b434846d-a8ea-4eec-a759-e5bd74ca12e8\") " Oct 30 06:10:29.814349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814322 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:29.814414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814341 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:29.814678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.814653 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:29.815346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.815326 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.815438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.815348 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.815438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.815360 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.815438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.815386 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.815438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.815399 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.817233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.817194 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:29.817351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.817253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "kube-api-access-wfmwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:29.817439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.817423 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:29.817510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.817439 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo" pod "b434846d-a8ea-4eec-a759-e5bd74ca12e8" (UID: "b434846d-a8ea-4eec-a759-e5bd74ca12e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:29.904303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.904273 6171 generic.go:356] "Generic (PLEG): container finished" podID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerID="f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb" exitCode=0 Oct 30 06:10:29.904478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.904327 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" Oct 30 06:10:29.904478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.904371 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerDied","Data":"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb"} Oct 30 06:10:29.904478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.904432 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc" event={"ID":"b434846d-a8ea-4eec-a759-e5bd74ca12e8","Type":"ContainerDied","Data":"9ae96323026ba9cda980470b76c0104e7ee731d2f5b0b8a169bedd62841d23aa"} Oct 30 06:10:29.904478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.904462 6171 scope.go:122] "RemoveContainer" containerID="f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb" Oct 30 06:10:29.917541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.917514 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.917541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.917538 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfmwm\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-kube-api-access-wfmwm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.917749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.917550 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.917749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.917560 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b434846d-a8ea-4eec-a759-e5bd74ca12e8-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:29.918038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.918017 6171 scope.go:122] "RemoveContainer" containerID="20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.962607 6171 scope.go:122] "RemoveContainer" containerID="f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:29.963998 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb\": container with ID starting with f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb not found: ID does not exist" containerID="f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.964058 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb"} err="failed to get container status \"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb\": rpc error: code = NotFound desc = could not find container \"f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb\": container with ID starting with f8be6150f02f8c87eaf663f790886d73e18fea9ff39e5eb306b357be2d526cfb not found: ID does not exist" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.964088 6171 scope.go:122] "RemoveContainer" containerID="20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:29.964482 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742\": container with ID starting with 20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742 not found: ID does not exist" containerID="20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742" Oct 30 06:10:29.988943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.964511 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742"} err="failed to get container status \"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742\": rpc error: code = NotFound desc = could not find container \"20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742\": container with ID starting with 20445f91d5429b08f564d713912d0682fce8143e4900375d6bc275cdb7bf7742 not found: ID does not exist" Oct 30 06:10:29.996377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:29.996304 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 06:10:30.002905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:30.002873 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-m8dcc"] Oct 30 06:10:31.680861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:31.680824 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" path="/var/lib/kubelet/pods/b434846d-a8ea-4eec-a759-e5bd74ca12e8/volumes" Oct 30 06:10:32.107228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.107194 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:10:32.107536 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.107492 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" containerID="cri-o://4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d" gracePeriod=30 Oct 30 06:10:32.118324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.118297 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:10:32.118521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.118502 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" containerID="cri-o://718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32" gracePeriod=30 Oct 30 06:10:32.697082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697026 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4"] Oct 30 06:10:32.697902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697885 6171 cpu_manager.go:414] "containerMap: removing container" podUID="96a70b3d-16f8-4765-acf8-35bb6d3a56f7" containerName="revoke-keys" Oct 30 06:10:32.697950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697903 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96a70b3d-16f8-4765-acf8-35bb6d3a56f7" containerName="revoke-keys" Oct 30 06:10:32.697950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697938 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.697950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697943 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697953 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697958 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697974 6171 cpu_manager.go:414] "containerMap: removing container" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697978 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697992 6171 cpu_manager.go:414] "containerMap: removing container" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.697997 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698010 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" Oct 30 06:10:32.698039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698015 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698170 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="79ff11b1-4490-4b67-b1fe-2b319f67f239" containerName="istio-proxy" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698184 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="33f0043c-6834-4bf4-88a3-fdd1662b51ff" containerName="istio-proxy" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698192 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffc9eee7-3498-464d-bff5-82b4d48a1360" containerName="main" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698201 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="96a70b3d-16f8-4765-acf8-35bb6d3a56f7" containerName="revoke-keys" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698216 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.698342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.698221 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b434846d-a8ea-4eec-a759-e5bd74ca12e8" containerName="istio-proxy" Oct 30 06:10:32.703898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.703872 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.720727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.720701 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4"] Oct 30 06:10:32.764093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.764034 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:32.764290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.764149 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:10:32.804277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.804244 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.804442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.804295 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tgx\" (UniqueName: \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.804442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.804418 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.906033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.906000 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.906286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.906092 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tgx\" (UniqueName: \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.906286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.906127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.906768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.906747 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.908731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.908712 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:32.931384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:32.931360 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tgx\" (UniqueName: \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\") pod \"maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:33.020088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:33.020030 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:33.371320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:33.371268 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4"] Oct 30 06:10:33.947624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:33.947525 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" event={"ID":"3ed252f8-38c4-4a75-8b27-86cd297a7a74","Type":"ContainerStarted","Data":"b1975ba5bec4ebd230e55c20fffa877573dc52c6c5f64807946bfb0015f90eb1"} Oct 30 06:10:33.947624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:33.947599 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" event={"ID":"3ed252f8-38c4-4a75-8b27-86cd297a7a74","Type":"ContainerStarted","Data":"fe28cb22f311e4f9c51b4fae25d4092fb6a5bf221f2bff765cf1dc1725cd5ea0"} Oct 30 06:10:33.994480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:33.994424 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" podStartSLOduration=1.994410275 podStartE2EDuration="1.994410275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:33.992365208 +0000 UTC m=+2292.432742403" watchObservedRunningTime="2026-10-30 06:10:33.994410275 +0000 UTC m=+2292.434787468" Oct 30 06:10:34.778281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.778247 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:34.778463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.778297 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:34.778712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.778685 6171 patch_prober.go:29] interesting pod/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:34.778829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.778724 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:34.960407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.960370 6171 generic.go:356] "Generic (PLEG): container finished" podID="3ed252f8-38c4-4a75-8b27-86cd297a7a74" containerID="b1975ba5bec4ebd230e55c20fffa877573dc52c6c5f64807946bfb0015f90eb1" exitCode=0 Oct 30 06:10:34.960807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:34.960454 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" event={"ID":"3ed252f8-38c4-4a75-8b27-86cd297a7a74","Type":"ContainerDied","Data":"b1975ba5bec4ebd230e55c20fffa877573dc52c6c5f64807946bfb0015f90eb1"} Oct 30 06:10:36.141354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.141326 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:36.282863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.282829 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\" (UniqueName: \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\") pod \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " Oct 30 06:10:36.283119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.282890 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\") pod \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " Oct 30 06:10:36.283207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.283185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\") pod \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\" (UID: \"3ed252f8-38c4-4a75-8b27-86cd297a7a74\") " Oct 30 06:10:36.283564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.283531 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca" pod "3ed252f8-38c4-4a75-8b27-86cd297a7a74" (UID: "3ed252f8-38c4-4a75-8b27-86cd297a7a74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:36.284237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.284215 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:36.285307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.285284 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca" pod "3ed252f8-38c4-4a75-8b27-86cd297a7a74" (UID: "3ed252f8-38c4-4a75-8b27-86cd297a7a74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:36.285604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.285583 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx" pod "3ed252f8-38c4-4a75-8b27-86cd297a7a74" (UID: "3ed252f8-38c4-4a75-8b27-86cd297a7a74"). InnerVolumeSpecName "kube-api-access-65tgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:36.385283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.385252 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65tgx\" (UniqueName: \"kubernetes.io/projected/3ed252f8-38c4-4a75-8b27-86cd297a7a74-kube-api-access-65tgx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:36.385283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.385277 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3ed252f8-38c4-4a75-8b27-86cd297a7a74-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:36.980180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.980150 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" Oct 30 06:10:36.980414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.980193 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4" event={"ID":"3ed252f8-38c4-4a75-8b27-86cd297a7a74","Type":"ContainerDied","Data":"fe28cb22f311e4f9c51b4fae25d4092fb6a5bf221f2bff765cf1dc1725cd5ea0"} Oct 30 06:10:36.980414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:36.980253 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe28cb22f311e4f9c51b4fae25d4092fb6a5bf221f2bff765cf1dc1725cd5ea0" Oct 30 06:10:37.412729 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.412706 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:10:37.456289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.456267 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:10:37.517599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517513 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.517599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517571 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.517833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517665 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.517833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517713 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.517833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517797 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517862 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.518010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517867 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.518010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517925 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517956 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.517988 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518078 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.518343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518109 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.518343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518120 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.518343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518242 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518324 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518352 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518392 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.518682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518479 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.518682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518510 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\") pod \"ad831650-4a65-447c-9145-035bcf85c782\" (UID: \"ad831650-4a65-447c-9145-035bcf85c782\") " Oct 30 06:10:37.518682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.518545 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\") pod \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\" (UID: \"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a\") " Oct 30 06:10:37.519713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.519571 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.519713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.519596 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.519713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.519615 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:37.519947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.519801 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.520017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.519978 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:37.520421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.520241 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.520598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.520569 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.520598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.520581 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.520792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.520623 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.520863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.520803 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.525608 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "kube-api-access-xgl86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.526783 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.526851 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.527221 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "kube-api-access-x2gnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.527331 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.527407 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy" pod "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" (UID: "80946ee4-ec79-48b2-9ae5-cb78f1c4c26a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.529229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.527477 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:37.529982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.529288 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy" pod "ad831650-4a65-447c-9145-035bcf85c782" (UID: "ad831650-4a65-447c-9145-035bcf85c782"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:37.620934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620898 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgl86\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-kube-api-access-xgl86\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.620934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620934 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620949 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620964 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620977 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.620992 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621004 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621016 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2gnc\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-kube-api-access-x2gnc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621032 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad831650-4a65-447c-9145-035bcf85c782-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621063 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad831650-4a65-447c-9145-035bcf85c782-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621075 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621089 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621101 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621113 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad831650-4a65-447c-9145-035bcf85c782-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621126 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad831650-4a65-447c-9145-035bcf85c782-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.621170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.621138 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:37.990326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.990294 6171 generic.go:356] "Generic (PLEG): container finished" podID="ad831650-4a65-447c-9145-035bcf85c782" containerID="4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d" exitCode=0 Oct 30 06:10:37.990512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.990347 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" Oct 30 06:10:37.990512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.990381 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerDied","Data":"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d"} Oct 30 06:10:37.990512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.990432 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8" event={"ID":"ad831650-4a65-447c-9145-035bcf85c782","Type":"ContainerDied","Data":"9d7adef96824a639cd417d8e67e9af14b74bcaefe27fb1847e0c46666f6fe72e"} Oct 30 06:10:37.990512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.990465 6171 scope.go:122] "RemoveContainer" containerID="4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d" Oct 30 06:10:37.992095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.992034 6171 generic.go:356] "Generic (PLEG): container finished" podID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerID="718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32" exitCode=0 Oct 30 06:10:37.992198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.992090 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" event={"ID":"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a","Type":"ContainerDied","Data":"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32"} Oct 30 06:10:37.992198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.992122 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" Oct 30 06:10:37.992198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:37.992134 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl" event={"ID":"80946ee4-ec79-48b2-9ae5-cb78f1c4c26a","Type":"ContainerDied","Data":"359adeeca51674ebaf957529dd31871aef6afa99413efc71dd7985203baeb729"} Oct 30 06:10:38.003021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.003001 6171 scope.go:122] "RemoveContainer" containerID="50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394" Oct 30 06:10:38.014434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.014410 6171 scope.go:122] "RemoveContainer" containerID="4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d" Oct 30 06:10:38.014729 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:38.014709 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d\": container with ID starting with 4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d not found: ID does not exist" containerID="4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d" Oct 30 06:10:38.014836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.014740 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d"} err="failed to get container status \"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d\": rpc error: code = NotFound desc = could not find container \"4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d\": container with ID starting with 4ad9c37a1dec253621423143a6052e2b75b55086704eca2c0406af511df6483d not found: ID does not exist" Oct 30 06:10:38.014836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.014762 6171 scope.go:122] "RemoveContainer" containerID="50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394" Oct 30 06:10:38.015026 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:38.015008 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394\": container with ID starting with 50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394 not found: ID does not exist" containerID="50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394" Oct 30 06:10:38.015100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.015032 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394"} err="failed to get container status \"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394\": rpc error: code = NotFound desc = could not find container \"50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394\": container with ID starting with 50df4e4ca3ccd6bcc210ffacf38b949f9ae5f4db546fbf5670466f8b6d7f5394 not found: ID does not exist" Oct 30 06:10:38.015100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.015069 6171 scope.go:122] "RemoveContainer" containerID="718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32" Oct 30 06:10:38.028320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.028303 6171 scope.go:122] "RemoveContainer" containerID="718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32" Oct 30 06:10:38.028575 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:38.028556 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32\": container with ID starting with 718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32 not found: ID does not exist" containerID="718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32" Oct 30 06:10:38.028654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.028583 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32"} err="failed to get container status \"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32\": rpc error: code = NotFound desc = could not find container \"718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32\": container with ID starting with 718dbc28947aeb6a5a07a792e791feffb4a90d12e6cb3cf00fdfb334d1acaa32 not found: ID does not exist" Oct 30 06:10:38.068480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.068458 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:10:38.082066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.082015 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-mgdvl"] Oct 30 06:10:38.109732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.109707 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:10:38.114306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.114285 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c1d09f-openshift-default-67b97686fc-wscv8"] Oct 30 06:10:38.469569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:38.469492 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:10:39.674704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:39.674671 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" path="/var/lib/kubelet/pods/80946ee4-ec79-48b2-9ae5-cb78f1c4c26a/volumes" Oct 30 06:10:39.675224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:39.675208 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad831650-4a65-447c-9145-035bcf85c782" path="/var/lib/kubelet/pods/ad831650-4a65-447c-9145-035bcf85c782/volumes" Oct 30 06:10:42.844617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.844586 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:10:42.845488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845468 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:42.845557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845499 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:42.845557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845512 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3ed252f8-38c4-4a75-8b27-86cd297a7a74" containerName="revoke-keys" Oct 30 06:10:42.845557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845518 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ed252f8-38c4-4a75-8b27-86cd297a7a74" containerName="revoke-keys" Oct 30 06:10:42.845557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845543 6171 cpu_manager.go:414] "containerMap: removing container" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" Oct 30 06:10:42.845557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845548 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" Oct 30 06:10:42.845718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845692 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:42.845718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845702 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ed252f8-38c4-4a75-8b27-86cd297a7a74" containerName="revoke-keys" Oct 30 06:10:42.845718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.845714 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="80946ee4-ec79-48b2-9ae5-cb78f1c4c26a" containerName="istio-proxy" Oct 30 06:10:42.851224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.851194 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:42.869691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.869662 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w1-24db22-serving-cert\"" Oct 30 06:10:42.872610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.872585 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:10:42.906345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.906309 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:42.906522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.906467 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sd9j\" (UniqueName: \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:42.906606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.906583 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:42.906670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:42.906655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.007781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.007746 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sd9j\" (UniqueName: \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.007959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.007841 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.007959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.007897 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.008107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.008088 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.008659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.008638 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.010582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.010558 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.010831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.010809 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.068083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.055754 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sd9j\" (UniqueName: \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\") pod \"maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.167781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.167688 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:43.500695 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:43.500668 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8bc4bd5_c2ee_4e58_87c0_66fd3b9264a6.slice/crio-3aadbd20ef9744f9f88127e1074c8a93851b921ca40eb91ebfef4ce4330b3983 WatchSource:0}: Error finding container 3aadbd20ef9744f9f88127e1074c8a93851b921ca40eb91ebfef4ce4330b3983: Status 404 returned error can't find the container with id 3aadbd20ef9744f9f88127e1074c8a93851b921ca40eb91ebfef4ce4330b3983 Oct 30 06:10:43.501210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:43.501184 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:10:44.129305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:44.129266 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" event={"ID":"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6","Type":"ContainerStarted","Data":"e87dd7fdc8950867d75bc75c0234805aa29957cd0cb54d6d5a9c331fdb1fde96"} Oct 30 06:10:44.129705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:44.129332 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" event={"ID":"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6","Type":"ContainerStarted","Data":"3aadbd20ef9744f9f88127e1074c8a93851b921ca40eb91ebfef4ce4330b3983"} Oct 30 06:10:44.129705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:44.129359 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:44.203070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:44.202933 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" podStartSLOduration=1.8736109669999998 podStartE2EDuration="2.2029115s" totalImagesPullingTime="329.300533ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:44.186702605 +0000 UTC m=+2302.627079800" watchObservedRunningTime="2026-10-30 06:10:44.2029115 +0000 UTC m=+2302.643288698" Oct 30 06:10:46.950943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.950915 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:10:46.989231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.989203 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:10:46.990055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990029 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:46.990120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990061 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:46.990120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990108 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" Oct 30 06:10:46.990120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990113 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" Oct 30 06:10:46.990259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990249 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerName="payload-pre-processing" Oct 30 06:10:46.990296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.990267 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad831650-4a65-447c-9145-035bcf85c782" containerName="istio-proxy" Oct 30 06:10:46.994199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:46.994180 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.018796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.018761 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-cc33d3\"/\"kube-root-ca.crt\"" Oct 30 06:10:47.019004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.018979 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-cc33d3\"/\"openshift-service-ca.crt\"" Oct 30 06:10:47.025644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.025613 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-cc33d3\"/\"default-dockercfg-j5pbp\"" Oct 30 06:10:47.026380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.026352 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-cc33d3\"/\"sub-test-model-cc33d3-kserve-self-signed-certs\"" Oct 30 06:10:47.026933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.026908 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:10:47.031387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.031360 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\" (UniqueName: \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\") pod \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " Oct 30 06:10:47.031585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.031566 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\") pod \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\" (UID: \"e9ccb28b-0899-4d2d-8f64-a66407b67dee\") " Oct 30 06:10:47.032260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032235 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.032383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032300 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.032462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032384 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr4l7\" (UniqueName: \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.032462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032452 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.032590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.032737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.032718 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.039814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.039482 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume" pod "e9ccb28b-0899-4d2d-8f64-a66407b67dee" (UID: "e9ccb28b-0899-4d2d-8f64-a66407b67dee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:47.049579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.049548 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv" pod "e9ccb28b-0899-4d2d-8f64-a66407b67dee" (UID: "e9ccb28b-0899-4d2d-8f64-a66407b67dee"). InnerVolumeSpecName "kube-api-access-z6rbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:47.133721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.133684 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.133922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.133817 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.133922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.133867 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.133932 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr4l7\" (UniqueName: \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134079 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134148 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134175 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134199 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.134409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134389 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9ccb28b-0899-4d2d-8f64-a66407b67dee-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:47.134470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134429 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6rbv\" (UniqueName: \"kubernetes.io/projected/e9ccb28b-0899-4d2d-8f64-a66407b67dee-kube-api-access-z6rbv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:10:47.134527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.134467 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.136399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.136374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.136674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.136657 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.157349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.157319 6171 generic.go:356] "Generic (PLEG): container finished" podID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" containerID="17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946" exitCode=0 Oct 30 06:10:47.157470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.157384 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" Oct 30 06:10:47.157470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.157407 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" event={"ID":"e9ccb28b-0899-4d2d-8f64-a66407b67dee","Type":"ContainerDied","Data":"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946"} Oct 30 06:10:47.157582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.157485 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql" event={"ID":"e9ccb28b-0899-4d2d-8f64-a66407b67dee","Type":"ContainerDied","Data":"e48bb76769069dfec26f96e751a1ea3f5ec8d5eceaf614ee36e1998f7dc206cf"} Oct 30 06:10:47.157582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.157512 6171 scope.go:122] "RemoveContainer" containerID="17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946" Oct 30 06:10:47.159953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.159927 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr4l7\" (UniqueName: \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\") pod \"sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.175834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.175809 6171 scope.go:122] "RemoveContainer" containerID="17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946" Oct 30 06:10:47.176306 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:47.176285 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946\": container with ID starting with 17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946 not found: ID does not exist" containerID="17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946" Oct 30 06:10:47.176378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.176313 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946"} err="failed to get container status \"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946\": rpc error: code = NotFound desc = could not find container \"17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946\": container with ID starting with 17b4aefe1f322bb850a6a23d94f5a88dbec8584787358f808bf0b8282b67e946 not found: ID does not exist" Oct 30 06:10:47.221232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.221210 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:10:47.231087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.231012 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c1d09f-5b9887dcc5-lgnql"] Oct 30 06:10:47.279487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.279459 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz"] Oct 30 06:10:47.286943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.286923 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7df9fa-bl5dz"] Oct 30 06:10:47.332586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.331394 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:47.685151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.685113 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96a70b3d-16f8-4765-acf8-35bb6d3a56f7" path="/var/lib/kubelet/pods/96a70b3d-16f8-4765-acf8-35bb6d3a56f7/volumes" Oct 30 06:10:47.685875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.685853 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9ccb28b-0899-4d2d-8f64-a66407b67dee" path="/var/lib/kubelet/pods/e9ccb28b-0899-4d2d-8f64-a66407b67dee/volumes" Oct 30 06:10:47.751228 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:47.751202 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1d70593_fbfb_42ec_acb0_429cd5952483.slice/crio-883096d99e81a1a0c985d42e6f9888f03d14ec3eef2ac69202e38a06a9def847 WatchSource:0}: Error finding container 883096d99e81a1a0c985d42e6f9888f03d14ec3eef2ac69202e38a06a9def847: Status 404 returned error can't find the container with id 883096d99e81a1a0c985d42e6f9888f03d14ec3eef2ac69202e38a06a9def847 Oct 30 06:10:47.752094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:47.752075 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:10:48.169368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.169283 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" event={"ID":"c1d70593-fbfb-42ec-acb0-429cd5952483","Type":"ContainerStarted","Data":"ca60b2edd7a28dff2d66bbabb75db63d22d3ab11a98b3b1b70849c2de274e93d"} Oct 30 06:10:48.169368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.169332 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" event={"ID":"c1d70593-fbfb-42ec-acb0-429cd5952483","Type":"ContainerStarted","Data":"883096d99e81a1a0c985d42e6f9888f03d14ec3eef2ac69202e38a06a9def847"} Oct 30 06:10:48.215833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.215786 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" podStartSLOduration=2.215770756 podStartE2EDuration="2.215770756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:48.215019631 +0000 UTC m=+2306.655396823" watchObservedRunningTime="2026-10-30 06:10:48.215770756 +0000 UTC m=+2306.656147949" Oct 30 06:10:48.292794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.292764 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:10:48.297868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.297831 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.315669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.315648 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-815776-openshift-default-dockercfg-wzpnm\"" Oct 30 06:10:48.380892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.374281 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:48.387820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.387778 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:10:48.483901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.483874 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484017 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484130 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484252 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484280 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484312 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484476 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484539 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwrx\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.484668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.484642 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.585998 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586031 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586070 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586157 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586224 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwrx\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586256 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586308 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586372 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586444 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586483 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586483 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.586895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.586867 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.587140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.587076 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.587140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.587096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.588944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.588919 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.589111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.589012 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.613793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.613637 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.615330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.615305 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwrx\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\") pod \"e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:48.912821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:48.912744 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:49.323783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:49.323758 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:10:49.325127 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:10:49.325103 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38fb093e_587f_4e18_bd9d_5b9b8b04f3ef.slice/crio-34878a20142f3175238bd5e5e57542e2b6211b92e3770b932ce2ab475a3f69b9 WatchSource:0}: Error finding container 34878a20142f3175238bd5e5e57542e2b6211b92e3770b932ce2ab475a3f69b9: Status 404 returned error can't find the container with id 34878a20142f3175238bd5e5e57542e2b6211b92e3770b932ce2ab475a3f69b9 Oct 30 06:10:49.327761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:49.327727 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:49.327821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:49.327795 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:49.327864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:49.327824 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:10:50.143886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.143859 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:10:50.190775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.190733 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" event={"ID":"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef","Type":"ContainerStarted","Data":"20fa3093f19c70e1acba6f485ba09d48fea0a1ede570b4f2da45ff0a96f458fa"} Oct 30 06:10:50.190973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.190794 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" event={"ID":"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef","Type":"ContainerStarted","Data":"34878a20142f3175238bd5e5e57542e2b6211b92e3770b932ce2ab475a3f69b9"} Oct 30 06:10:50.279029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.274297 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podStartSLOduration=2.274274862 podStartE2EDuration="2.274274862s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:50.243685936 +0000 UTC m=+2308.684063130" watchObservedRunningTime="2026-10-30 06:10:50.274274862 +0000 UTC m=+2308.714652059" Oct 30 06:10:50.915007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.913388 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:50.919448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.919411 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:50.919619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:50.919488 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:51.913224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:51.913187 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:51.913423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:51.913251 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:52.918909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:52.918312 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:52.918909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:52.918385 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:53.917080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:53.914509 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:53.917080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:53.914585 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:54.913198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:54.913165 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:54.913697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:54.913224 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:55.655470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:55.655424 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:10:55.656020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:55.655977 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" containerID="cri-o://f6829214cdd4484a25ede93ec4afe72a155160801cc18cd187865dbfc1afb8e5" gracePeriod=30 Oct 30 06:10:55.933885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:55.927169 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:55.933885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:55.927232 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:56.251126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.229949 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l"] Oct 30 06:10:56.507352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.507251 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l"] Oct 30 06:10:56.528229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.517971 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.557216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.529675 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:10:56.557216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.546371 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:10:56.648071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.626381 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:56.648840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.648801 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": context deadline exceeded" Oct 30 06:10:56.762736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.729981 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr6m9\" (UniqueName: \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.762736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.730247 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.778557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.778518 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.856800 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:10:56.856758 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:10:56.880898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.880864 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.881324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.881295 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr6m9\" (UniqueName: \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.881631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.881610 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.886448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.883939 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.907488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.907459 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:56.917730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.917704 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" start-of-body= Oct 30 06:10:56.917945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.917917 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.50:15021/healthz/ready\": dial tcp 10.133.1.50:15021: connect: connection refused" Oct 30 06:10:56.931410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:56.927088 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr6m9\" (UniqueName: \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\") pod \"maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:57.104453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.104362 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:10:57.335376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.333908 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:57.354548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.354476 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:57.532093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.531959 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l"] Oct 30 06:10:57.842167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.842105 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" event={"ID":"766ad58b-d979-4a51-8ccc-ae9bd96f202b","Type":"ContainerStarted","Data":"a30babe189b2849abc9859bb293a69e203cb4478cf942014adcefb9f1e92fdd9"} Oct 30 06:10:57.863737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:57.863704 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:10:58.040099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.032254 6171 patch_prober.go:29] interesting pod/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:58.040099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.032318 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:58.237271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.233875 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:58.253276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.248252 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:58.253276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.253219 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:10:58.852626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.852589 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" event={"ID":"766ad58b-d979-4a51-8ccc-ae9bd96f202b","Type":"ContainerStarted","Data":"416691d959eee124d468bdfb62a4d7032162fc3d3c54d3dc3e7a3ee260b375be"} Oct 30 06:10:58.916594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:58.916542 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" podStartSLOduration=2.916527689 podStartE2EDuration="2.916527689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:58.913527805 +0000 UTC m=+2317.353904999" watchObservedRunningTime="2026-10-30 06:10:58.916527689 +0000 UTC m=+2317.356904882" Oct 30 06:10:59.864180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:59.864143 6171 generic.go:356] "Generic (PLEG): container finished" podID="766ad58b-d979-4a51-8ccc-ae9bd96f202b" containerID="416691d959eee124d468bdfb62a4d7032162fc3d3c54d3dc3e7a3ee260b375be" exitCode=0 Oct 30 06:10:59.864603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:59.864199 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" event={"ID":"766ad58b-d979-4a51-8ccc-ae9bd96f202b","Type":"ContainerDied","Data":"416691d959eee124d468bdfb62a4d7032162fc3d3c54d3dc3e7a3ee260b375be"} Oct 30 06:10:59.941748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:59.941711 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7"] Oct 30 06:10:59.946763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:59.946738 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:10:59.968358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:10:59.968311 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7"] Oct 30 06:11:00.038206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.038166 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.038544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.038521 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblgx\" (UniqueName: \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.038850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.038828 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.140218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.140125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.140605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.140578 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.140796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.140773 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblgx\" (UniqueName: \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.141379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.141353 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.143302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.143276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.168807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.168778 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblgx\" (UniqueName: \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\") pod \"maas-api-revoke-sub-a5a6380f-9shx7\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.265882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.265849 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:00.664211 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:00.664175 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5735c8ac_caa5_40b6_b466_499440aefdb8.slice/crio-eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0 WatchSource:0}: Error finding container eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0: Status 404 returned error can't find the container with id eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0 Oct 30 06:11:00.665490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.665464 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7"] Oct 30 06:11:00.922034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.921950 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" event={"ID":"5735c8ac-caa5-40b6-b466-499440aefdb8","Type":"ContainerStarted","Data":"eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0"} Oct 30 06:11:00.923921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.923893 6171 generic.go:356] "Generic (PLEG): container finished" podID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerID="f6829214cdd4484a25ede93ec4afe72a155160801cc18cd187865dbfc1afb8e5" exitCode=0 Oct 30 06:11:00.924316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.924293 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" event={"ID":"cfe3ebc4-95c4-43a2-8994-548791ba7cc9","Type":"ContainerDied","Data":"f6829214cdd4484a25ede93ec4afe72a155160801cc18cd187865dbfc1afb8e5"} Oct 30 06:11:00.938888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:00.938527 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:11:01.057469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057436 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.057654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057510 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.057654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057552 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.057654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057609 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.057810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057703 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057909 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.057937 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058134 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058171 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\") pod \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\" (UID: \"cfe3ebc4-95c4-43a2-8994-548791ba7cc9\") " Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058166 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058398 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058612 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058976 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.058992 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.059005 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.059239 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:01.060475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.060098 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.063537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.063507 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.064470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.064447 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:01.110073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.077448 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:01.110073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.078178 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f" pod "cfe3ebc4-95c4-43a2-8994-548791ba7cc9" (UID: "cfe3ebc4-95c4-43a2-8994-548791ba7cc9"). InnerVolumeSpecName "kube-api-access-r7l9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.160881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160846 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7l9f\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-kube-api-access-r7l9f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.160881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160875 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.160881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160887 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.161159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160895 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.161159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160903 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.161159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.160911 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfe3ebc4-95c4-43a2-8994-548791ba7cc9-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.252825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.252800 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:11:01.363469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.363438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\") pod \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " Oct 30 06:11:01.363685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.363551 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\" (UniqueName: \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\") pod \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " Oct 30 06:11:01.363752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.363691 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\") pod \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\" (UID: \"766ad58b-d979-4a51-8ccc-ae9bd96f202b\") " Oct 30 06:11:01.364186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.364148 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca" pod "766ad58b-d979-4a51-8ccc-ae9bd96f202b" (UID: "766ad58b-d979-4a51-8ccc-ae9bd96f202b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:01.364443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.364421 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.366177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.366153 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9" pod "766ad58b-d979-4a51-8ccc-ae9bd96f202b" (UID: "766ad58b-d979-4a51-8ccc-ae9bd96f202b"). InnerVolumeSpecName "kube-api-access-xr6m9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.366279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.366187 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca" pod "766ad58b-d979-4a51-8ccc-ae9bd96f202b" (UID: "766ad58b-d979-4a51-8ccc-ae9bd96f202b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:01.466107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.465993 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/766ad58b-d979-4a51-8ccc-ae9bd96f202b-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.466107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:01.466031 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr6m9\" (UniqueName: \"kubernetes.io/projected/766ad58b-d979-4a51-8ccc-ae9bd96f202b-kube-api-access-xr6m9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:02.046159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.046134 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" Oct 30 06:11:02.046655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.046141 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l" event={"ID":"766ad58b-d979-4a51-8ccc-ae9bd96f202b","Type":"ContainerDied","Data":"a30babe189b2849abc9859bb293a69e203cb4478cf942014adcefb9f1e92fdd9"} Oct 30 06:11:02.046756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.046683 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a30babe189b2849abc9859bb293a69e203cb4478cf942014adcefb9f1e92fdd9" Oct 30 06:11:02.048168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.048145 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" event={"ID":"5735c8ac-caa5-40b6-b466-499440aefdb8","Type":"ContainerStarted","Data":"fe96ab29b43d0717b26700310e0aea029b43ed34dd5c216003b9c97be0c3e9c2"} Oct 30 06:11:02.049715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.049694 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" event={"ID":"cfe3ebc4-95c4-43a2-8994-548791ba7cc9","Type":"ContainerDied","Data":"57ccccc5be6645c8b9847169287badf8f31a379c7e08ee5e53e0d449b96d2801"} Oct 30 06:11:02.049810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.049717 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg" Oct 30 06:11:02.049810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.049756 6171 scope.go:122] "RemoveContainer" containerID="f6829214cdd4484a25ede93ec4afe72a155160801cc18cd187865dbfc1afb8e5" Oct 30 06:11:02.149960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.149925 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:11:02.155061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:02.155009 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-7df9fa-openshift-default-b67f7d48f-jt6wg"] Oct 30 06:11:03.064925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.064889 6171 generic.go:356] "Generic (PLEG): container finished" podID="5735c8ac-caa5-40b6-b466-499440aefdb8" containerID="fe96ab29b43d0717b26700310e0aea029b43ed34dd5c216003b9c97be0c3e9c2" exitCode=0 Oct 30 06:11:03.065455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.064977 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" event={"ID":"5735c8ac-caa5-40b6-b466-499440aefdb8","Type":"ContainerDied","Data":"fe96ab29b43d0717b26700310e0aea029b43ed34dd5c216003b9c97be0c3e9c2"} Oct 30 06:11:03.272716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.272659 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:03.394123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.382327 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\") pod \"5735c8ac-caa5-40b6-b466-499440aefdb8\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " Oct 30 06:11:03.394123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.382525 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\") pod \"5735c8ac-caa5-40b6-b466-499440aefdb8\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " Oct 30 06:11:03.394123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.382582 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\" (UniqueName: \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\") pod \"5735c8ac-caa5-40b6-b466-499440aefdb8\" (UID: \"5735c8ac-caa5-40b6-b466-499440aefdb8\") " Oct 30 06:11:03.422921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.422885 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca" pod "5735c8ac-caa5-40b6-b466-499440aefdb8" (UID: "5735c8ac-caa5-40b6-b466-499440aefdb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:03.463958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.460556 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca" pod "5735c8ac-caa5-40b6-b466-499440aefdb8" (UID: "5735c8ac-caa5-40b6-b466-499440aefdb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:03.502114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.488426 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:03.502114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.488462 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5735c8ac-caa5-40b6-b466-499440aefdb8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:03.532067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.528473 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx" pod "5735c8ac-caa5-40b6-b466-499440aefdb8" (UID: "5735c8ac-caa5-40b6-b466-499440aefdb8"). InnerVolumeSpecName "kube-api-access-nblgx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:03.590363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.590321 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nblgx\" (UniqueName: \"kubernetes.io/projected/5735c8ac-caa5-40b6-b466-499440aefdb8-kube-api-access-nblgx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:03.674681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:03.674588 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" path="/var/lib/kubelet/pods/cfe3ebc4-95c4-43a2-8994-548791ba7cc9/volumes" Oct 30 06:11:04.080228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:04.080199 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" Oct 30 06:11:04.080764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:04.080238 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7" event={"ID":"5735c8ac-caa5-40b6-b466-499440aefdb8","Type":"ContainerDied","Data":"eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0"} Oct 30 06:11:04.080764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:04.080285 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea230e9ee8a20bd9c3332b1dcb39402101d45066464fa4a96fb40cb99d917b0" Oct 30 06:11:04.642294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:04.642187 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:11:04.642294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:04.642277 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:06.548258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:06.548211 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:11:06.584564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:06.584527 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:11:08.196424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.196392 6171 generic.go:356] "Generic (PLEG): container finished" podID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerID="f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8" exitCode=0 Oct 30 06:11:08.196869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.196463 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerDied","Data":"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8"} Oct 30 06:11:08.197003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.196984 6171 scope.go:122] "RemoveContainer" containerID="f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8" Oct 30 06:11:08.198959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.198928 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:08.199077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.199004 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:08.199077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:08.199038 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:09.216193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:09.216156 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerStarted","Data":"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00"} Oct 30 06:11:09.375290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:09.375205 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:11:09.375498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:09.375460 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:09.375574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:09.375528 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:10.076088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.076018 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:11:10.076328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.076293 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" podUID="6ef33713-6170-4f22-9560-7f04b24c419f" containerName="maas-api" containerID="cri-o://6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2" gracePeriod=30 Oct 30 06:11:10.375983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.375446 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:10.375983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.375521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:10.552072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.501672 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l"] Oct 30 06:11:10.845074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.839519 6171 cpu_manager.go:414] "containerMap: removing container" podUID="766ad58b-d979-4a51-8ccc-ae9bd96f202b" containerName="revoke-keys" Oct 30 06:11:10.861672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.861634 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="766ad58b-d979-4a51-8ccc-ae9bd96f202b" containerName="revoke-keys" Oct 30 06:11:10.861962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.861941 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" Oct 30 06:11:10.862118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.862101 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" Oct 30 06:11:10.862247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.862237 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5735c8ac-caa5-40b6-b466-499440aefdb8" containerName="revoke-keys" Oct 30 06:11:10.862336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.862323 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5735c8ac-caa5-40b6-b466-499440aefdb8" containerName="revoke-keys" Oct 30 06:11:10.862774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.862744 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="766ad58b-d979-4a51-8ccc-ae9bd96f202b" containerName="revoke-keys" Oct 30 06:11:10.862952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.862935 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfe3ebc4-95c4-43a2-8994-548791ba7cc9" containerName="istio-proxy" Oct 30 06:11:10.863100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.863087 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5735c8ac-caa5-40b6-b466-499440aefdb8" containerName="revoke-keys" Oct 30 06:11:10.931461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.923703 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l"] Oct 30 06:11:10.931461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.924103 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:10.956063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.945010 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:10.956063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.945149 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:11:10.965265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.964912 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4"] Oct 30 06:11:10.967813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.967365 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:11:10.973420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:10.973399 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-62316c-bbmf4"] Oct 30 06:11:11.024898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.024867 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\" (UniqueName: \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\") pod \"6ef33713-6170-4f22-9560-7f04b24c419f\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " Oct 30 06:11:11.025106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.025024 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\") pod \"6ef33713-6170-4f22-9560-7f04b24c419f\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " Oct 30 06:11:11.025188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.025161 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\") pod \"6ef33713-6170-4f22-9560-7f04b24c419f\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " Oct 30 06:11:11.025337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.025316 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\") pod \"6ef33713-6170-4f22-9560-7f04b24c419f\" (UID: \"6ef33713-6170-4f22-9560-7f04b24c419f\") " Oct 30 06:11:11.027776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.027742 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.028091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028066 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.028262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7n88v\" (UniqueName: \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.028472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028450 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls" pod "6ef33713-6170-4f22-9560-7f04b24c419f" (UID: "6ef33713-6170-4f22-9560-7f04b24c419f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:11.028565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028484 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca" pod "6ef33713-6170-4f22-9560-7f04b24c419f" (UID: "6ef33713-6170-4f22-9560-7f04b24c419f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:11.028627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028579 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs" pod "6ef33713-6170-4f22-9560-7f04b24c419f" (UID: "6ef33713-6170-4f22-9560-7f04b24c419f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:11.028736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.028710 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm" pod "6ef33713-6170-4f22-9560-7f04b24c419f" (UID: "6ef33713-6170-4f22-9560-7f04b24c419f"). InnerVolumeSpecName "kube-api-access-5tdqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:11.130773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.130687 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7n88v\" (UniqueName: \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.132847 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.133189 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.133406 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ef33713-6170-4f22-9560-7f04b24c419f-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.133428 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.133443 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tdqm\" (UniqueName: \"kubernetes.io/projected/6ef33713-6170-4f22-9560-7f04b24c419f-kube-api-access-5tdqm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.133456 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef33713-6170-4f22-9560-7f04b24c419f-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:11.139304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.134510 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.142472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.142173 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.157808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.157775 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7n88v\" (UniqueName: \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\") pod \"maas-api-revoke-sub-74d02af5-7ch6l\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.237979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.237943 6171 generic.go:356] "Generic (PLEG): container finished" podID="6ef33713-6170-4f22-9560-7f04b24c419f" containerID="6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2" exitCode=0 Oct 30 06:11:11.238210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.238015 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" Oct 30 06:11:11.238210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.238071 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" event={"ID":"6ef33713-6170-4f22-9560-7f04b24c419f","Type":"ContainerDied","Data":"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2"} Oct 30 06:11:11.238210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.238170 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz" event={"ID":"6ef33713-6170-4f22-9560-7f04b24c419f","Type":"ContainerDied","Data":"2cdf199e74113babc2fa3c4cf21c2e629364d48c011dcd90b368c4cbceb46fb7"} Oct 30 06:11:11.238210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.238194 6171 scope.go:122] "RemoveContainer" containerID="6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2" Oct 30 06:11:11.251198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.251179 6171 scope.go:122] "RemoveContainer" containerID="6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2" Oct 30 06:11:11.251456 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:11.251440 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2\": container with ID starting with 6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2 not found: ID does not exist" containerID="6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2" Oct 30 06:11:11.251503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.251464 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2"} err="failed to get container status \"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2\": rpc error: code = NotFound desc = could not find container \"6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2\": container with ID starting with 6d781d5a75793de81434aba10724a49cd0f5c35085671b29c97408b3c3ced7a2 not found: ID does not exist" Oct 30 06:11:11.296377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.296342 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:11.312624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.312597 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:11:11.318397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.318372 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-62316c-54b8877b4f-wdmzz"] Oct 30 06:11:11.376142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.376117 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:11.376442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.376178 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:11.548920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.548878 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" containerID="cri-o://b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e" gracePeriod=25 Oct 30 06:11:11.602071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.601117 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" containerID="cri-o://28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b" gracePeriod=25 Oct 30 06:11:11.622491 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:11.622459 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda59416ab_f8c7_499c_87bf_857da99648fd.slice/crio-217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb WatchSource:0}: Error finding container 217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb: Status 404 returned error can't find the container with id 217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb Oct 30 06:11:11.623605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.623583 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l"] Oct 30 06:11:11.675808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.675779 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ed252f8-38c4-4a75-8b27-86cd297a7a74" path="/var/lib/kubelet/pods/3ed252f8-38c4-4a75-8b27-86cd297a7a74/volumes" Oct 30 06:11:11.676364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:11.676346 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef33713-6170-4f22-9560-7f04b24c419f" path="/var/lib/kubelet/pods/6ef33713-6170-4f22-9560-7f04b24c419f/volumes" Oct 30 06:11:12.250039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:12.249998 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" event={"ID":"a59416ab-f8c7-499c-87bf-857da99648fd","Type":"ContainerStarted","Data":"424915c527a18a3c037936ab6d257fe260a80a2322a9654587242da6070212de"} Oct 30 06:11:12.250252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:12.250102 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" event={"ID":"a59416ab-f8c7-499c-87bf-857da99648fd","Type":"ContainerStarted","Data":"217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb"} Oct 30 06:11:12.309164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:12.309030 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" podStartSLOduration=2.309009777 podStartE2EDuration="2.309009777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:12.297468516 +0000 UTC m=+2330.737845713" watchObservedRunningTime="2026-10-30 06:11:12.309009777 +0000 UTC m=+2330.749386972" Oct 30 06:11:12.375409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:12.375375 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:12.375574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:12.375425 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:13.261737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.261703 6171 generic.go:356] "Generic (PLEG): container finished" podID="a59416ab-f8c7-499c-87bf-857da99648fd" containerID="424915c527a18a3c037936ab6d257fe260a80a2322a9654587242da6070212de" exitCode=0 Oct 30 06:11:13.262198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.261781 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" event={"ID":"a59416ab-f8c7-499c-87bf-857da99648fd","Type":"ContainerDied","Data":"424915c527a18a3c037936ab6d257fe260a80a2322a9654587242da6070212de"} Oct 30 06:11:13.375490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.375457 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:13.375682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.375521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:13.878073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.871531 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:11:13.878073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:13.871772 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" containerName="maas-api" containerID="cri-o://3f28ad3192a9c920b318619bbbd8cca06e8187fcd0f2e42f005d14b8d6c3c997" gracePeriod=30 Oct 30 06:11:14.623205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.579011 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" start-of-body= Oct 30 06:11:14.623205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.579093 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.35:15021/healthz/ready\": dial tcp 10.133.1.35:15021: connect: connection refused" Oct 30 06:11:14.635454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.596017 6171 generic.go:356] "Generic (PLEG): container finished" podID="f5215668-1f63-40b2-9ddb-85095076ef29" containerID="3f28ad3192a9c920b318619bbbd8cca06e8187fcd0f2e42f005d14b8d6c3c997" exitCode=0 Oct 30 06:11:14.635454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.596134 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" event={"ID":"f5215668-1f63-40b2-9ddb-85095076ef29","Type":"ContainerDied","Data":"3f28ad3192a9c920b318619bbbd8cca06e8187fcd0f2e42f005d14b8d6c3c997"} Oct 30 06:11:14.674825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.674794 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:11:14.818272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.818238 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l"] Oct 30 06:11:14.825599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.825572 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d64c9-xfb8l"] Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.893159 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\" (UniqueName: \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\") pod \"f5215668-1f63-40b2-9ddb-85095076ef29\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.893214 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\") pod \"f5215668-1f63-40b2-9ddb-85095076ef29\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.893368 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\") pod \"f5215668-1f63-40b2-9ddb-85095076ef29\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.893579 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") pod \"f5215668-1f63-40b2-9ddb-85095076ef29\" (UID: \"f5215668-1f63-40b2-9ddb-85095076ef29\") " Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.899791 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf" pod "f5215668-1f63-40b2-9ddb-85095076ef29" (UID: "f5215668-1f63-40b2-9ddb-85095076ef29"). InnerVolumeSpecName "kube-api-access-hwgsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:14.906601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.900159 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca" pod "f5215668-1f63-40b2-9ddb-85095076ef29" (UID: "f5215668-1f63-40b2-9ddb-85095076ef29"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:14.925033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.924829 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs" pod "f5215668-1f63-40b2-9ddb-85095076ef29" (UID: "f5215668-1f63-40b2-9ddb-85095076ef29"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:14.931896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.931854 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls" pod "f5215668-1f63-40b2-9ddb-85095076ef29" (UID: "f5215668-1f63-40b2-9ddb-85095076ef29"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:14.950177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.950150 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:14.994797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.994764 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\") pod \"a59416ab-f8c7-499c-87bf-857da99648fd\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " Oct 30 06:11:14.994992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.994924 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\" (UniqueName: \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\") pod \"a59416ab-f8c7-499c-87bf-857da99648fd\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " Oct 30 06:11:14.995109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995068 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\") pod \"a59416ab-f8c7-499c-87bf-857da99648fd\" (UID: \"a59416ab-f8c7-499c-87bf-857da99648fd\") " Oct 30 06:11:14.995180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995113 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca" pod "a59416ab-f8c7-499c-87bf-857da99648fd" (UID: "a59416ab-f8c7-499c-87bf-857da99648fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:14.995676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995596 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:14.995676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995620 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:14.995676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995631 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwgsf\" (UniqueName: \"kubernetes.io/projected/f5215668-1f63-40b2-9ddb-85095076ef29-kube-api-access-hwgsf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:14.995676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995641 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5215668-1f63-40b2-9ddb-85095076ef29-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:14.995676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.995650 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f5215668-1f63-40b2-9ddb-85095076ef29-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:14.997208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.997181 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v" pod "a59416ab-f8c7-499c-87bf-857da99648fd" (UID: "a59416ab-f8c7-499c-87bf-857da99648fd"). InnerVolumeSpecName "kube-api-access-7n88v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:14.997338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:14.997234 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca" pod "a59416ab-f8c7-499c-87bf-857da99648fd" (UID: "a59416ab-f8c7-499c-87bf-857da99648fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:15.098344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.098294 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7n88v\" (UniqueName: \"kubernetes.io/projected/a59416ab-f8c7-499c-87bf-857da99648fd-kube-api-access-7n88v\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:15.098344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.098343 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a59416ab-f8c7-499c-87bf-857da99648fd-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:15.379739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.379712 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:11:15.380099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.380077 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:11:15.380846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.380828 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:11:15.639914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.639838 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" Oct 30 06:11:15.639914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.639866 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" event={"ID":"f5215668-1f63-40b2-9ddb-85095076ef29","Type":"ContainerDied","Data":"b64549941d64ec94d4e4d642973e0146579c591f7c520304fd7a1058b942c34d"} Oct 30 06:11:15.640465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.639933 6171 scope.go:122] "RemoveContainer" containerID="3f28ad3192a9c920b318619bbbd8cca06e8187fcd0f2e42f005d14b8d6c3c997" Oct 30 06:11:15.641619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.641593 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" event={"ID":"a59416ab-f8c7-499c-87bf-857da99648fd","Type":"ContainerDied","Data":"217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb"} Oct 30 06:11:15.641819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.641796 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="217a33aed399405cc0596565d2584d7a7b446fd5288eb2a4cd8c550d716e78bb" Oct 30 06:11:15.641819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.641600 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l" Oct 30 06:11:15.673910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.673884 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="766ad58b-d979-4a51-8ccc-ae9bd96f202b" path="/var/lib/kubelet/pods/766ad58b-d979-4a51-8ccc-ae9bd96f202b/volumes" Oct 30 06:11:15.716489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.716467 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:11:15.723195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:15.723175 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz"] Oct 30 06:11:17.577437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:17.577398 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-0d64c9-bd65566d-sqbwz" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.7:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:11:17.674136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:17.674101 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" path="/var/lib/kubelet/pods/f5215668-1f63-40b2-9ddb-85095076ef29/volumes" Oct 30 06:11:18.340498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:18.340452 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:18.349668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:18.349628 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:18.983457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:18.983424 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:11:18.983879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:18.983667 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" containerID="cri-o://394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2" gracePeriod=30 Oct 30 06:11:20.596321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.596286 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd"] Oct 30 06:11:20.597145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597129 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6ef33713-6170-4f22-9560-7f04b24c419f" containerName="maas-api" Oct 30 06:11:20.597204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597146 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef33713-6170-4f22-9560-7f04b24c419f" containerName="maas-api" Oct 30 06:11:20.597204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597157 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a59416ab-f8c7-499c-87bf-857da99648fd" containerName="revoke-keys" Oct 30 06:11:20.597204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597163 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a59416ab-f8c7-499c-87bf-857da99648fd" containerName="revoke-keys" Oct 30 06:11:20.597204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597181 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" containerName="maas-api" Oct 30 06:11:20.597204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597186 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" containerName="maas-api" Oct 30 06:11:20.597365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597335 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5215668-1f63-40b2-9ddb-85095076ef29" containerName="maas-api" Oct 30 06:11:20.597365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597354 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef33713-6170-4f22-9560-7f04b24c419f" containerName="maas-api" Oct 30 06:11:20.597365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.597360 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a59416ab-f8c7-499c-87bf-857da99648fd" containerName="revoke-keys" Oct 30 06:11:20.602078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.602057 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.624191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.624163 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd"] Oct 30 06:11:20.630205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.630184 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:20.631436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.630914 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:11:20.644391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.644367 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:11:20.649627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.649604 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.667248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.667221 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5b1e\"/\"kube-root-ca.crt\"" Oct 30 06:11:20.668551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.668532 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5b1e\"/\"openshift-service-ca.crt\"" Oct 30 06:11:20.682537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.682513 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:11:20.691957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.691928 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7lph\" (UniqueName: \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.692191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692070 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.692191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692183 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.692390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692216 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt9j6\" (UniqueName: \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.692390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692268 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.692390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692369 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.692654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692451 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.692654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692530 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.692654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.692599 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.723754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.723725 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5b1e\"/\"auth-test-model-6b5b1e-kserve-self-signed-certs\"" Oct 30 06:11:20.723754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.723725 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6b5b1e\"/\"default-dockercfg-x95n8\"" Oct 30 06:11:20.794294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794266 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.794400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794301 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt9j6\" (UniqueName: \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.794400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794342 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794432 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794505 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794583 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.794656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794643 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794699 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7lph\" (UniqueName: \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794801 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.794945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.794914 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.795131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.795106 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.795307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.795284 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.795422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.795388 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.797059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.797022 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.797152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.797139 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.797248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.797232 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.820484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.820463 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7lph\" (UniqueName: \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\") pod \"auth-test-model-6b5b1e-kserve-86bb474b5-j64xf\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:20.820923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.820905 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt9j6\" (UniqueName: \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\") pod \"maas-api-revoke-sub-c610016d-vmpjd\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.948141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.940317 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:20.982069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:20.976428 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:21.356811 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:21.356772 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod024caeea_1005_42e4_88a3_3da505b79aa7.slice/crio-9117e5c50d6d24d93ec7f8c0ad70f27390ba1c2d98ec376f18b455378596593a WatchSource:0}: Error finding container 9117e5c50d6d24d93ec7f8c0ad70f27390ba1c2d98ec376f18b455378596593a: Status 404 returned error can't find the container with id 9117e5c50d6d24d93ec7f8c0ad70f27390ba1c2d98ec376f18b455378596593a Oct 30 06:11:21.358105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.358075 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:11:21.369590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.369564 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd"] Oct 30 06:11:21.370990 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:21.370968 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0917f3_cd24_43f9_b95f_4b8149066f68.slice/crio-c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b WatchSource:0}: Error finding container c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b: Status 404 returned error can't find the container with id c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b Oct 30 06:11:21.710893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.710861 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" event={"ID":"1a0917f3-cd24-43f9-b95f-4b8149066f68","Type":"ContainerStarted","Data":"6bcaed7e7fe414f59ff4f9c64b1f5df438d4a8c44d563b7a50a53c3e26a427ae"} Oct 30 06:11:21.711271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.710922 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" event={"ID":"1a0917f3-cd24-43f9-b95f-4b8149066f68","Type":"ContainerStarted","Data":"c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b"} Oct 30 06:11:21.712743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.712715 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" event={"ID":"024caeea-1005-42e4-88a3-3da505b79aa7","Type":"ContainerStarted","Data":"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33"} Oct 30 06:11:21.712889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.712754 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" event={"ID":"024caeea-1005-42e4-88a3-3da505b79aa7","Type":"ContainerStarted","Data":"9117e5c50d6d24d93ec7f8c0ad70f27390ba1c2d98ec376f18b455378596593a"} Oct 30 06:11:21.767750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.767680 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" podStartSLOduration=1.767663218 podStartE2EDuration="1.767663218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:21.758172485 +0000 UTC m=+2340.198549678" watchObservedRunningTime="2026-10-30 06:11:21.767663218 +0000 UTC m=+2340.208040412" Oct 30 06:11:21.809155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.809107 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" podStartSLOduration=1.8090910679999999 podStartE2EDuration="1.809091068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:21.80829769 +0000 UTC m=+2340.248674883" watchObservedRunningTime="2026-10-30 06:11:21.809091068 +0000 UTC m=+2340.249468260" Oct 30 06:11:21.977165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:21.977127 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:22.724373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:22.724330 6171 generic.go:356] "Generic (PLEG): container finished" podID="1a0917f3-cd24-43f9-b95f-4b8149066f68" containerID="6bcaed7e7fe414f59ff4f9c64b1f5df438d4a8c44d563b7a50a53c3e26a427ae" exitCode=0 Oct 30 06:11:22.724789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:22.724370 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" event={"ID":"1a0917f3-cd24-43f9-b95f-4b8149066f68","Type":"ContainerDied","Data":"6bcaed7e7fe414f59ff4f9c64b1f5df438d4a8c44d563b7a50a53c3e26a427ae"} Oct 30 06:11:22.955392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:22.955357 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:11:22.955616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:22.955588 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" containerID="cri-o://744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9" gracePeriod=30 Oct 30 06:11:24.125080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.125002 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:24.243079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.239645 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\" (UniqueName: \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\") pod \"1a0917f3-cd24-43f9-b95f-4b8149066f68\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " Oct 30 06:11:24.282066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.266619 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\") pod \"1a0917f3-cd24-43f9-b95f-4b8149066f68\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " Oct 30 06:11:24.282066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.266786 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\") pod \"1a0917f3-cd24-43f9-b95f-4b8149066f68\" (UID: \"1a0917f3-cd24-43f9-b95f-4b8149066f68\") " Oct 30 06:11:24.291228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.287126 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca" pod "1a0917f3-cd24-43f9-b95f-4b8149066f68" (UID: "1a0917f3-cd24-43f9-b95f-4b8149066f68"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:24.292231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.292168 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6" pod "1a0917f3-cd24-43f9-b95f-4b8149066f68" (UID: "1a0917f3-cd24-43f9-b95f-4b8149066f68"). InnerVolumeSpecName "kube-api-access-lt9j6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:24.317646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.312555 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca" pod "1a0917f3-cd24-43f9-b95f-4b8149066f68" (UID: "1a0917f3-cd24-43f9-b95f-4b8149066f68"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:24.337014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.336982 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:11:24.368675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.368812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368694 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.368812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368755 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.368812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368785 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.368995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368893 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.368995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.368930 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.369145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369000 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.369145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369061 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.369145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369072 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.369145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369078 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.369337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369255 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.369337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369294 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:24.369447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369353 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\") pod \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\" (UID: \"a6eca143-8c4b-43c7-9b09-027c4dfd12e6\") " Oct 30 06:11:24.369668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.369500 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.370240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370205 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370234 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt9j6\" (UniqueName: \"kubernetes.io/projected/1a0917f3-cd24-43f9-b95f-4b8149066f68-kube-api-access-lt9j6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370252 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370267 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370281 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370294 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370307 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.370462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.370320 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a0917f3-cd24-43f9-b95f-4b8149066f68-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.371291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.371270 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "kube-api-access-zqzgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:24.371379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.371363 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:24.371425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.371407 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:24.371487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.371473 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo" pod "a6eca143-8c4b-43c7-9b09-027c4dfd12e6" (UID: "a6eca143-8c4b-43c7-9b09-027c4dfd12e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:24.472421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.472393 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqzgc\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-kube-api-access-zqzgc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.472556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.472431 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.472556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.472449 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.472556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.472461 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6eca143-8c4b-43c7-9b09-027c4dfd12e6-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:24.864780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.864690 6171 generic.go:356] "Generic (PLEG): container finished" podID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerID="394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2" exitCode=0 Oct 30 06:11:24.864780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.864761 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" Oct 30 06:11:24.865001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.864771 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerDied","Data":"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2"} Oct 30 06:11:24.865001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.864851 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b" event={"ID":"a6eca143-8c4b-43c7-9b09-027c4dfd12e6","Type":"ContainerDied","Data":"6d51791e6431efed2bf3e70c1d7728d87f11c404330ebddf8679c8cb33976f36"} Oct 30 06:11:24.865001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.864875 6171 scope.go:122] "RemoveContainer" containerID="394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2" Oct 30 06:11:24.866511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.866491 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" Oct 30 06:11:24.866615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.866508 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd" event={"ID":"1a0917f3-cd24-43f9-b95f-4b8149066f68","Type":"ContainerDied","Data":"c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b"} Oct 30 06:11:24.866615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.866541 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62d5101613db96555c57c45d2e8159920ae09bec5fe3af9ce768aa48e68256b" Oct 30 06:11:24.878671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.878600 6171 scope.go:122] "RemoveContainer" containerID="ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477" Oct 30 06:11:24.891387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.891368 6171 scope.go:122] "RemoveContainer" containerID="394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2" Oct 30 06:11:24.891668 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:24.891649 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2\": container with ID starting with 394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2 not found: ID does not exist" containerID="394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2" Oct 30 06:11:24.891769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.891675 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2"} err="failed to get container status \"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2\": rpc error: code = NotFound desc = could not find container \"394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2\": container with ID starting with 394d9bb93e9b5f0cf256a47ba6c4be3b573542cdbcdf1f6f32c966609571c6a2 not found: ID does not exist" Oct 30 06:11:24.891769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.891691 6171 scope.go:122] "RemoveContainer" containerID="ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477" Oct 30 06:11:24.891961 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:24.891932 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477\": container with ID starting with ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477 not found: ID does not exist" containerID="ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477" Oct 30 06:11:24.892010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.891973 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477"} err="failed to get container status \"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477\": rpc error: code = NotFound desc = could not find container \"ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477\": container with ID starting with ee0dbf5781a261acecf8330a3742b9c441865607388dc7d17f5c9098e4dea477 not found: ID does not exist" Oct 30 06:11:24.926111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.926082 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:11:24.940410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:24.940384 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-62316c-openshift-default-96996f9c9-q792b"] Oct 30 06:11:25.675710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:25.675677 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" path="/var/lib/kubelet/pods/a6eca143-8c4b-43c7-9b09-027c4dfd12e6/volumes" Oct 30 06:11:26.382130 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:26.382098 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:11:27.704082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.704037 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:11:27.705114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705091 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.705114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705114 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.705326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705135 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.705326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705142 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.705326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705158 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1a0917f3-cd24-43f9-b95f-4b8149066f68" containerName="revoke-keys" Oct 30 06:11:27.705326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705166 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0917f3-cd24-43f9-b95f-4b8149066f68" containerName="revoke-keys" Oct 30 06:11:27.705624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705427 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.705624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705468 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0917f3-cd24-43f9-b95f-4b8149066f68" containerName="revoke-keys" Oct 30 06:11:27.705624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.705481 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6eca143-8c4b-43c7-9b09-027c4dfd12e6" containerName="istio-proxy" Oct 30 06:11:27.726432 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.721580 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.736253 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.736231 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:11:27.743159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.743140 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-0f980e-openshift-default-dockercfg-sk7nx\"" Oct 30 06:11:27.833880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.833852 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.833971 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f98kv\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834116 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834168 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834323 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834360 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834546 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.834639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.834613 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943541 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943687 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f98kv\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943790 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943842 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943918 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.943958 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.958481 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.958614 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.958866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.958657 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.959566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.959244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.963084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.959881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.963084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.960558 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.963084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.960791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:27.963084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:27.962551 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.053134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.053099 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f98kv\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.053640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.053612 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.085576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.085534 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.114405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.114374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.316903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.316875 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:11:28.338694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.337259 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:28.338694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.338110 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:11:28.350640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.350604 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:28.472781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.472750 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.472972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.472844 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.472972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.472915 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473036 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473114 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473188 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473192 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.473309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473239 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473280 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473458 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.473479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473475 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\") pod \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\" (UID: \"d26fcb88-4030-4e78-ad07-7e8e33fbbe24\") " Oct 30 06:11:28.473627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473475 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.473762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473743 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.473824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.473679 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:28.474502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.474460 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.474502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.474493 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.474658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.474507 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.474658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.474519 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.474658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.474535 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.475913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.475888 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:28.476082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.476030 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:28.476390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.476363 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "kube-api-access-xqbs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:28.476752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.476727 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy" pod "d26fcb88-4030-4e78-ad07-7e8e33fbbe24" (UID: "d26fcb88-4030-4e78-ad07-7e8e33fbbe24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:28.576013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.575986 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.576013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.576013 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.576013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.576022 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.576243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.576032 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqbs9\" (UniqueName: \"kubernetes.io/projected/d26fcb88-4030-4e78-ad07-7e8e33fbbe24-kube-api-access-xqbs9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:28.689462 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:28.689436 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb9ebe3a_c3de_4555_8405_fb741f440345.slice/crio-0e7100fd7ae2b47cc0deb8055a681ac343c72ea49be3ecf8500dec81d9cbc749 WatchSource:0}: Error finding container 0e7100fd7ae2b47cc0deb8055a681ac343c72ea49be3ecf8500dec81d9cbc749: Status 404 returned error can't find the container with id 0e7100fd7ae2b47cc0deb8055a681ac343c72ea49be3ecf8500dec81d9cbc749 Oct 30 06:11:28.690624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.690601 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:11:28.692660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.692617 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:28.692743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.692705 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:28.692780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.692748 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:28.923093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.923030 6171 generic.go:356] "Generic (PLEG): container finished" podID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerID="744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9" exitCode=0 Oct 30 06:11:28.923534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.923113 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" Oct 30 06:11:28.923534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.923113 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerDied","Data":"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9"} Oct 30 06:11:28.923534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.923173 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9" event={"ID":"d26fcb88-4030-4e78-ad07-7e8e33fbbe24","Type":"ContainerDied","Data":"7eacb3e44ad1e1ae6c27854086c2e7a6e8f311d7c0173eddc383d2cd56aaadd3"} Oct 30 06:11:28.923534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.923206 6171 scope.go:122] "RemoveContainer" containerID="744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9" Oct 30 06:11:28.925510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.925477 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerStarted","Data":"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176"} Oct 30 06:11:28.925625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.925538 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerStarted","Data":"0e7100fd7ae2b47cc0deb8055a681ac343c72ea49be3ecf8500dec81d9cbc749"} Oct 30 06:11:28.935822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.935804 6171 scope.go:122] "RemoveContainer" containerID="84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153" Oct 30 06:11:28.946707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.946683 6171 scope.go:122] "RemoveContainer" containerID="744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9" Oct 30 06:11:28.947018 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:28.946992 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9\": container with ID starting with 744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9 not found: ID does not exist" containerID="744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9" Oct 30 06:11:28.947127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.947030 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9"} err="failed to get container status \"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9\": rpc error: code = NotFound desc = could not find container \"744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9\": container with ID starting with 744fcc8c5ac337d96231417581c6bc4e9ceab690b37cc89cbcde73cc1cc31ba9 not found: ID does not exist" Oct 30 06:11:28.947127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.947096 6171 scope.go:122] "RemoveContainer" containerID="84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153" Oct 30 06:11:28.947423 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:28.947400 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153\": container with ID starting with 84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153 not found: ID does not exist" containerID="84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153" Oct 30 06:11:28.947484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.947430 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153"} err="failed to get container status \"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153\": rpc error: code = NotFound desc = could not find container \"84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153\": container with ID starting with 84d225053490bf6f9ccfd1c61320e49a07a96ab9a768b869ecf04ab7fb8d0153 not found: ID does not exist" Oct 30 06:11:28.991416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:28.991319 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podStartSLOduration=1.991305262 podStartE2EDuration="1.991305262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:28.989078001 +0000 UTC m=+2347.429455194" watchObservedRunningTime="2026-10-30 06:11:28.991305262 +0000 UTC m=+2347.431682455" Oct 30 06:11:29.033591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.033563 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:11:29.038824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.038797 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d64c9-openshift-default-585f89bc6f-lztt9"] Oct 30 06:11:29.043028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.043007 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw"] Oct 30 06:11:29.047543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.047520 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db9e71ee-bxccw"] Oct 30 06:11:29.351196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.351113 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:29.358396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.358374 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:29.674715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.674642 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc" path="/var/lib/kubelet/pods/bc7dacb2-bb78-49f0-8fe0-a0ebd939eafc/volumes" Oct 30 06:11:29.675207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.675192 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" path="/var/lib/kubelet/pods/d26fcb88-4030-4e78-ad07-7e8e33fbbe24/volumes" Oct 30 06:11:29.936971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.936892 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:29.938010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:29.937987 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:11:30.977763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:30.977735 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:30.990436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:30.990414 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:31.920316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.920292 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:11:31.924727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.924709 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:11:31.956836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.956807 6171 generic.go:356] "Generic (PLEG): container finished" podID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerID="b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e" exitCode=0 Oct 30 06:11:31.956951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.956848 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" Oct 30 06:11:31.956951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.956894 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" event={"ID":"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d","Type":"ContainerDied","Data":"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e"} Oct 30 06:11:31.957146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.956951 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27" event={"ID":"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d","Type":"ContainerDied","Data":"868c3385d9d33495c469aabc430b20e8a48dbb2bf702456ab7192a887029a84b"} Oct 30 06:11:31.957146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.956981 6171 scope.go:122] "RemoveContainer" containerID="b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e" Oct 30 06:11:31.959800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.959774 6171 generic.go:356] "Generic (PLEG): container finished" podID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerID="28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b" exitCode=0 Oct 30 06:11:31.959910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.959804 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" event={"ID":"213bc107-2f1f-4a90-bd30-29ed691231c8","Type":"ContainerDied","Data":"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b"} Oct 30 06:11:31.959910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.959827 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" Oct 30 06:11:31.959910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.959847 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47" event={"ID":"213bc107-2f1f-4a90-bd30-29ed691231c8","Type":"ContainerDied","Data":"bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9"} Oct 30 06:11:31.972188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.972167 6171 scope.go:122] "RemoveContainer" containerID="b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e" Oct 30 06:11:31.972666 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:31.972506 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e\": container with ID starting with b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e not found: ID does not exist" containerID="b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e" Oct 30 06:11:31.972666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.972542 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e"} err="failed to get container status \"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e\": rpc error: code = NotFound desc = could not find container \"b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e\": container with ID starting with b2436b262eaec93ab7ed50b3db3fa625d63f1a47fdb59d15d7cc977e1925200e not found: ID does not exist" Oct 30 06:11:31.972666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.972563 6171 scope.go:122] "RemoveContainer" containerID="28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b" Oct 30 06:11:31.973725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.973538 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:11:31.984144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.984123 6171 scope.go:122] "RemoveContainer" containerID="28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b" Oct 30 06:11:31.984494 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:31.984456 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b\": container with ID starting with 28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b not found: ID does not exist" containerID="28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b" Oct 30 06:11:31.984494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:31.984483 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b"} err="failed to get container status \"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b\": rpc error: code = NotFound desc = could not find container \"28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b\": container with ID starting with 28ddbab4a2ee045d354ac561a38d36ca6f51c8e5f0778628fe3eb4fde8068d2b not found: ID does not exist" Oct 30 06:11:32.047937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.047903 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\" (UniqueName: \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\") pod \"213bc107-2f1f-4a90-bd30-29ed691231c8\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " Oct 30 06:11:32.048135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.048106 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\") pod \"213bc107-2f1f-4a90-bd30-29ed691231c8\" (UID: \"213bc107-2f1f-4a90-bd30-29ed691231c8\") " Oct 30 06:11:32.048217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.048164 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\") pod \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " Oct 30 06:11:32.048366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.048345 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\" (UniqueName: \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\") pod \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\" (UID: \"6349b3ad-2c77-4dbd-996a-3c2091c1cf2d\") " Oct 30 06:11:32.048684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.048652 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume" pod "213bc107-2f1f-4a90-bd30-29ed691231c8" (UID: "213bc107-2f1f-4a90-bd30-29ed691231c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:32.049085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.049034 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume" pod "6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" (UID: "6349b3ad-2c77-4dbd-996a-3c2091c1cf2d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:32.049185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.049138 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/213bc107-2f1f-4a90-bd30-29ed691231c8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:32.051535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.051508 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6" pod "6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" (UID: "6349b3ad-2c77-4dbd-996a-3c2091c1cf2d"). InnerVolumeSpecName "kube-api-access-qzvh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:32.051674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.051655 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r" pod "213bc107-2f1f-4a90-bd30-29ed691231c8" (UID: "213bc107-2f1f-4a90-bd30-29ed691231c8"). InnerVolumeSpecName "kube-api-access-hlp6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:32.150591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.150503 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:32.150591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.150528 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzvh6\" (UniqueName: \"kubernetes.io/projected/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d-kube-api-access-qzvh6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:32.150591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.150553 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlp6r\" (UniqueName: \"kubernetes.io/projected/213bc107-2f1f-4a90-bd30-29ed691231c8-kube-api-access-hlp6r\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:32.317763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.317735 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:11:32.323559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.323534 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-0d64c9-65ff998c55-6bx27"] Oct 30 06:11:32.341550 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:32.341520 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6349b3ad_2c77_4dbd_996a_3c2091c1cf2d.slice/crio-868c3385d9d33495c469aabc430b20e8a48dbb2bf702456ab7192a887029a84b\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213bc107_2f1f_4a90_bd30_29ed691231c8.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213bc107_2f1f_4a90_bd30_29ed691231c8.slice/crio-bdb014633e9d7b3e34de2e25ddd4d10e5d3eaee6acbed801ac6d4ad4979d22b9\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6349b3ad_2c77_4dbd_996a_3c2091c1cf2d.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:11:32.359603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.359578 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:11:32.364234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:32.364213 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-0d64c9-7bcf489987-jxs47"] Oct 30 06:11:33.173009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.172978 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:11:33.173870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173851 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.173918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173872 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.173918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173887 6171 cpu_manager.go:414] "containerMap: removing container" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" Oct 30 06:11:33.173918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173892 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" Oct 30 06:11:33.173918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173912 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.173918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173916 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173935 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.173940 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.174083 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.174094 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" containerName="payload-pre-processing" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.174102 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d26fcb88-4030-4e78-ad07-7e8e33fbbe24" containerName="istio-proxy" Oct 30 06:11:33.174112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.174110 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" containerName="payload-processing" Oct 30 06:11:33.179362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.179342 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.197354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.197329 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:11:33.203639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.203620 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0f980e\"" Oct 30 06:11:33.203809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.203620 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0f980e-dockercfg-v5cq2\"" Oct 30 06:11:33.466518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.466230 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dklk5\" (UniqueName: \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.466518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.466437 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.496127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.488169 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:11:33.496886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.496835 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:11:33.497022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.496951 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.567729 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.567696 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dklk5\" (UniqueName: \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.567865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.567837 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.568717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.568696 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.593304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.593277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dklk5\" (UniqueName: \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\") pod \"payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.668982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.668951 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9rls\" (UniqueName: \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.669140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.668988 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.673819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.673770 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="213bc107-2f1f-4a90-bd30-29ed691231c8" path="/var/lib/kubelet/pods/213bc107-2f1f-4a90-bd30-29ed691231c8/volumes" Oct 30 06:11:33.674306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.674284 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6349b3ad-2c77-4dbd-996a-3c2091c1cf2d" path="/var/lib/kubelet/pods/6349b3ad-2c77-4dbd-996a-3c2091c1cf2d/volumes" Oct 30 06:11:33.769852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.769830 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9rls\" (UniqueName: \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.769958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.769861 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.770614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.770596 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.795024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.794993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9rls\" (UniqueName: \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\") pod \"payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:33.796804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.796787 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:33.814917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:33.814897 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:34.119807 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:34.119777 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fc9606_370f_4fcb_b16c_5ec12af271b8.slice/crio-e62526a0a1cdcae9f4c0e4240a617cc30faffdc9abd22732f8ac24684c6762d5 WatchSource:0}: Error finding container e62526a0a1cdcae9f4c0e4240a617cc30faffdc9abd22732f8ac24684c6762d5: Status 404 returned error can't find the container with id e62526a0a1cdcae9f4c0e4240a617cc30faffdc9abd22732f8ac24684c6762d5 Oct 30 06:11:34.120913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:34.120887 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:11:34.153981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:34.153955 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:11:34.163482 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:34.163453 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5188865_8c7e_493e_806f_c98453ea8e94.slice/crio-fd1d0a243c4ed39f6da92d49c488e3c6b8c6dde27142efc0b8c2dd0b06f89824 WatchSource:0}: Error finding container fd1d0a243c4ed39f6da92d49c488e3c6b8c6dde27142efc0b8c2dd0b06f89824: Status 404 returned error can't find the container with id fd1d0a243c4ed39f6da92d49c488e3c6b8c6dde27142efc0b8c2dd0b06f89824 Oct 30 06:11:35.014840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.014800 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" event={"ID":"89fc9606-370f-4fcb-b16c-5ec12af271b8","Type":"ContainerStarted","Data":"2d4f261b5342d60be847d5d6140f2ea8d81243aa1f5bc5492c6c4771e2591464"} Oct 30 06:11:35.015360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.014869 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" event={"ID":"89fc9606-370f-4fcb-b16c-5ec12af271b8","Type":"ContainerStarted","Data":"e62526a0a1cdcae9f4c0e4240a617cc30faffdc9abd22732f8ac24684c6762d5"} Oct 30 06:11:35.015360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.014901 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:35.016703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.016676 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" event={"ID":"f5188865-8c7e-493e-806f-c98453ea8e94","Type":"ContainerStarted","Data":"d5fe23e5b4cb0cc28197a73097b170ee2fef39d81d4939936557a87201edb495"} Oct 30 06:11:35.016824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.016725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" event={"ID":"f5188865-8c7e-493e-806f-c98453ea8e94","Type":"ContainerStarted","Data":"fd1d0a243c4ed39f6da92d49c488e3c6b8c6dde27142efc0b8c2dd0b06f89824"} Oct 30 06:11:35.016824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.016746 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:35.067645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.067599 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" podStartSLOduration=2.067585574 podStartE2EDuration="2.067585574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:35.063799978 +0000 UTC m=+2353.504177174" watchObservedRunningTime="2026-10-30 06:11:35.067585574 +0000 UTC m=+2353.507962767" Oct 30 06:11:35.114950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:35.114906 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" podStartSLOduration=2.114892095 podStartE2EDuration="2.114892095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:35.111779115 +0000 UTC m=+2353.552156309" watchObservedRunningTime="2026-10-30 06:11:35.114892095 +0000 UTC m=+2353.555269287" Oct 30 06:11:38.037527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.037491 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:11:38.042888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.042870 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.066177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.066150 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w5-91f42c-serving-cert\"" Oct 30 06:11:38.066344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.066285 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:11:38.194403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.194362 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.194584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.194411 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.194649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.194589 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxxtw\" (UniqueName: \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.194727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.194660 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.296374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.296267 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxxtw\" (UniqueName: \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.296685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.296376 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.296685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.296485 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.296685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.296521 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.297892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.297865 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.300402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.300372 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.324109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.316716 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.326411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.326383 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxxtw\" (UniqueName: \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\") pod \"maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.364715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.364692 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:38.678427 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:38.678386 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0d3557_e4b6_40b6_bb8c_8f0ab9572196.slice/crio-9614e38d3904177a4beb0a21327071f5e6a693008e1209c7775fa50ec03b30fe WatchSource:0}: Error finding container 9614e38d3904177a4beb0a21327071f5e6a693008e1209c7775fa50ec03b30fe: Status 404 returned error can't find the container with id 9614e38d3904177a4beb0a21327071f5e6a693008e1209c7775fa50ec03b30fe Oct 30 06:11:38.678810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:38.678785 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:11:39.057778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:39.057751 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" event={"ID":"da0d3557-e4b6-40b6-bb8c-8f0ab9572196","Type":"ContainerStarted","Data":"9614e38d3904177a4beb0a21327071f5e6a693008e1209c7775fa50ec03b30fe"} Oct 30 06:11:40.036244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.036203 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l"] Oct 30 06:11:40.049003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.048976 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0e3355f-rsv2l"] Oct 30 06:11:40.068959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.068924 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" event={"ID":"da0d3557-e4b6-40b6-bb8c-8f0ab9572196","Type":"ContainerStarted","Data":"f0ada4bc562a5be6415dbc8f60934f62616617430898740bcbb62d7c5401297b"} Oct 30 06:11:40.069356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.069008 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:40.120743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.120694 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" podStartSLOduration=1.830074485 podStartE2EDuration="2.120680127s" totalImagesPullingTime="290.605642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:40.114991182 +0000 UTC m=+2358.555368375" watchObservedRunningTime="2026-10-30 06:11:40.120680127 +0000 UTC m=+2358.561057319" Oct 30 06:11:40.206688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.206651 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4"] Oct 30 06:11:40.211675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.211652 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.243884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.243850 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:11:40.244263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.244193 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:40.244937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.244904 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4"] Oct 30 06:11:40.433995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.433902 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.434174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.434095 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.434174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.434156 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkgq6\" (UniqueName: \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.535056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.535006 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.535230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.535129 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkgq6\" (UniqueName: \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.535306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.535244 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.535871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.535848 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.537873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.537849 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.562008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.561979 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkgq6\" (UniqueName: \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\") pod \"maas-api-revoke-sub-4431ec71-rs2h4\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:40.828232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:40.828193 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:41.143476 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:41.143445 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d91c81_6a94_48ed_9760_8d57f51870e9.slice/crio-6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398 WatchSource:0}: Error finding container 6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398: Status 404 returned error can't find the container with id 6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398 Oct 30 06:11:41.143848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:41.143827 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4"] Oct 30 06:11:41.675053 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:41.674971 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="135faeae-e50d-4ad5-859e-e4671c6c7c02" path="/var/lib/kubelet/pods/135faeae-e50d-4ad5-859e-e4671c6c7c02/volumes" Oct 30 06:11:42.090201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:42.090160 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" event={"ID":"08d91c81-6a94-48ed-9760-8d57f51870e9","Type":"ContainerStarted","Data":"e98fba4e0142483f3b598d4e1f26e42964357135e9acb6f99d2bfb5ed68494f9"} Oct 30 06:11:42.090381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:42.090226 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" event={"ID":"08d91c81-6a94-48ed-9760-8d57f51870e9","Type":"ContainerStarted","Data":"6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398"} Oct 30 06:11:42.136152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:42.136105 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" podStartSLOduration=2.136092605 podStartE2EDuration="2.136092605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:42.135149959 +0000 UTC m=+2360.575527144" watchObservedRunningTime="2026-10-30 06:11:42.136092605 +0000 UTC m=+2360.576469836" Oct 30 06:11:43.101255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:43.101222 6171 generic.go:356] "Generic (PLEG): container finished" podID="08d91c81-6a94-48ed-9760-8d57f51870e9" containerID="e98fba4e0142483f3b598d4e1f26e42964357135e9acb6f99d2bfb5ed68494f9" exitCode=0 Oct 30 06:11:43.101652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:43.101312 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" event={"ID":"08d91c81-6a94-48ed-9760-8d57f51870e9","Type":"ContainerDied","Data":"e98fba4e0142483f3b598d4e1f26e42964357135e9acb6f99d2bfb5ed68494f9"} Oct 30 06:11:44.288038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.288013 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:44.425737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.425662 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\" (UniqueName: \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\") pod \"08d91c81-6a94-48ed-9760-8d57f51870e9\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " Oct 30 06:11:44.425892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.425818 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\") pod \"08d91c81-6a94-48ed-9760-8d57f51870e9\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " Oct 30 06:11:44.425892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.425851 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\") pod \"08d91c81-6a94-48ed-9760-8d57f51870e9\" (UID: \"08d91c81-6a94-48ed-9760-8d57f51870e9\") " Oct 30 06:11:44.426274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.426246 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca" pod "08d91c81-6a94-48ed-9760-8d57f51870e9" (UID: "08d91c81-6a94-48ed-9760-8d57f51870e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:44.426577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.426496 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:44.427957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.427932 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6" pod "08d91c81-6a94-48ed-9760-8d57f51870e9" (UID: "08d91c81-6a94-48ed-9760-8d57f51870e9"). InnerVolumeSpecName "kube-api-access-qkgq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:44.428108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.427981 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca" pod "08d91c81-6a94-48ed-9760-8d57f51870e9" (UID: "08d91c81-6a94-48ed-9760-8d57f51870e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:44.528017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.527979 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkgq6\" (UniqueName: \"kubernetes.io/projected/08d91c81-6a94-48ed-9760-8d57f51870e9-kube-api-access-qkgq6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:44.528017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:44.528009 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08d91c81-6a94-48ed-9760-8d57f51870e9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:45.120100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:45.120072 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" Oct 30 06:11:45.120310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:45.120097 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4" event={"ID":"08d91c81-6a94-48ed-9760-8d57f51870e9","Type":"ContainerDied","Data":"6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398"} Oct 30 06:11:45.120310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:45.120172 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bfeee48712ba6a17583ffaed06ea3165d22c268a6a9ec64a72399580aef0398" Oct 30 06:11:46.031291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:46.031259 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:11:46.031737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:46.031719 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:11:46.085960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:46.085932 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:11:46.593403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:46.593351 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:11:51.772648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.772612 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9"] Oct 30 06:11:51.773512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.773496 6171 cpu_manager.go:414] "containerMap: removing container" podUID="08d91c81-6a94-48ed-9760-8d57f51870e9" containerName="revoke-keys" Oct 30 06:11:51.773565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.773512 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08d91c81-6a94-48ed-9760-8d57f51870e9" containerName="revoke-keys" Oct 30 06:11:51.773651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.773642 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="08d91c81-6a94-48ed-9760-8d57f51870e9" containerName="revoke-keys" Oct 30 06:11:51.777696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.777678 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.811379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.809825 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:51.811379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.809982 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:11:51.813841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.813809 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9"] Oct 30 06:11:51.866982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.866934 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.867184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.867010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.867317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.867295 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8pdv\" (UniqueName: \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.968545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.968503 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.968738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.968565 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.968847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.968833 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8pdv\" (UniqueName: \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.969340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.969312 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:51.971281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:51.971250 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:52.008011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:52.007978 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8pdv\" (UniqueName: \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\") pod \"maas-api-revoke-sub-5efbc29c-vxsl9\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:52.121657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:52.121567 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:52.459079 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:11:52.459023 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b101ad_1b9d_4616_9203_22ca69c49b3d.slice/crio-6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260 WatchSource:0}: Error finding container 6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260: Status 404 returned error can't find the container with id 6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260 Oct 30 06:11:52.459674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:52.459650 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9"] Oct 30 06:11:53.411138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:53.408166 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" event={"ID":"92b101ad-1b9d-4616-9203-22ca69c49b3d","Type":"ContainerStarted","Data":"18dbc86271dd813b01a71bf23c0ba66763ee9d68a0f87d43a413f618317b562b"} Oct 30 06:11:53.411138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:53.408238 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" event={"ID":"92b101ad-1b9d-4616-9203-22ca69c49b3d","Type":"ContainerStarted","Data":"6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260"} Oct 30 06:11:53.472410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:53.472330 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" podStartSLOduration=2.472313785 podStartE2EDuration="2.472313785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:53.468263957 +0000 UTC m=+2371.908641154" watchObservedRunningTime="2026-10-30 06:11:53.472313785 +0000 UTC m=+2371.912690977" Oct 30 06:11:54.421950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:54.421910 6171 generic.go:356] "Generic (PLEG): container finished" podID="92b101ad-1b9d-4616-9203-22ca69c49b3d" containerID="18dbc86271dd813b01a71bf23c0ba66763ee9d68a0f87d43a413f618317b562b" exitCode=0 Oct 30 06:11:54.422410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:54.422038 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" event={"ID":"92b101ad-1b9d-4616-9203-22ca69c49b3d","Type":"ContainerDied","Data":"18dbc86271dd813b01a71bf23c0ba66763ee9d68a0f87d43a413f618317b562b"} Oct 30 06:11:55.626190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.626160 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:55.795409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.795166 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\") pod \"92b101ad-1b9d-4616-9203-22ca69c49b3d\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " Oct 30 06:11:55.795409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.795371 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\" (UniqueName: \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\") pod \"92b101ad-1b9d-4616-9203-22ca69c49b3d\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " Oct 30 06:11:55.795675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.795432 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\") pod \"92b101ad-1b9d-4616-9203-22ca69c49b3d\" (UID: \"92b101ad-1b9d-4616-9203-22ca69c49b3d\") " Oct 30 06:11:55.796319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.796287 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca" pod "92b101ad-1b9d-4616-9203-22ca69c49b3d" (UID: "92b101ad-1b9d-4616-9203-22ca69c49b3d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:55.798906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.798867 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca" pod "92b101ad-1b9d-4616-9203-22ca69c49b3d" (UID: "92b101ad-1b9d-4616-9203-22ca69c49b3d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:55.798906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.798877 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv" pod "92b101ad-1b9d-4616-9203-22ca69c49b3d" (UID: "92b101ad-1b9d-4616-9203-22ca69c49b3d"). InnerVolumeSpecName "kube-api-access-s8pdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:55.898008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.897970 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:55.898008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.898002 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8pdv\" (UniqueName: \"kubernetes.io/projected/92b101ad-1b9d-4616-9203-22ca69c49b3d-kube-api-access-s8pdv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:55.898008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:55.898012 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b101ad-1b9d-4616-9203-22ca69c49b3d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:11:56.385453 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:11:56.385411 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:11:56.448559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:56.448519 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" event={"ID":"92b101ad-1b9d-4616-9203-22ca69c49b3d","Type":"ContainerDied","Data":"6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260"} Oct 30 06:11:56.448764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:56.448581 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6adced8cfdcdaef8aafc441f204ab0a0efbd3007918c924dea5cbededaf0f260" Oct 30 06:11:56.448764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:56.448525 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9" Oct 30 06:11:59.484840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.484802 6171 generic.go:356] "Generic (PLEG): container finished" podID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerID="939405f21115328c1b3a1001de04c79b481a9a8c1474879382f2e5606dd07805" exitCode=0 Oct 30 06:11:59.485324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.484887 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerDied","Data":"939405f21115328c1b3a1001de04c79b481a9a8c1474879382f2e5606dd07805"} Oct 30 06:11:59.485584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.485565 6171 scope.go:122] "RemoveContainer" containerID="939405f21115328c1b3a1001de04c79b481a9a8c1474879382f2e5606dd07805" Oct 30 06:11:59.487490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.487454 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:59.487646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.487528 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:11:59.487646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:11:59.487561 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:00.510069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:00.504700 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerStarted","Data":"3c4551783472c27a0885433564a0544a72797e1c4797f3d7639061b89690d9e3"} Oct 30 06:12:01.494071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:01.490385 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:12:01.503066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:01.499161 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 06:12:01.503066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:01.499239 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 06:12:02.074286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.074253 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q"] Oct 30 06:12:02.075214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.075192 6171 cpu_manager.go:414] "containerMap: removing container" podUID="92b101ad-1b9d-4616-9203-22ca69c49b3d" containerName="revoke-keys" Oct 30 06:12:02.075214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.075212 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b101ad-1b9d-4616-9203-22ca69c49b3d" containerName="revoke-keys" Oct 30 06:12:02.075403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.075342 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b101ad-1b9d-4616-9203-22ca69c49b3d" containerName="revoke-keys" Oct 30 06:12:02.079506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.079485 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.113449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.113411 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q"] Oct 30 06:12:02.115943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.115914 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:02.116401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.116378 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:12:02.230847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.230797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.231127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.231104 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9btcx\" (UniqueName: \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.231223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.231204 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.333156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.333001 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9btcx\" (UniqueName: \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.333156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.333125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.333424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.333209 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.333786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.333764 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.335853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.335829 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.361646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.361619 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9btcx\" (UniqueName: \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\") pod \"maas-api-revoke-sub-79b306a9-sgx9q\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.425538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.425495 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:02.507332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.487527 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 06:12:02.507332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.487591 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 06:12:02.773293 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:02.773258 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e88e9bf_6f7c_4b7d_bd9e_df554a49ed02.slice/crio-e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033 WatchSource:0}: Error finding container e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033: Status 404 returned error can't find the container with id e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033 Oct 30 06:12:02.774272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:02.774245 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q"] Oct 30 06:12:03.470134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.470097 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:12:03.497071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.495598 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 06:12:03.497071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.495661 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 06:12:03.553440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.553397 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" event={"ID":"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02","Type":"ContainerStarted","Data":"c7e214427a2923aa8e61f6398f1aa571bc38b0995273e9a30ffd3cc9c7e2c8ea"} Oct 30 06:12:03.553653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.553454 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" event={"ID":"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02","Type":"ContainerStarted","Data":"e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033"} Oct 30 06:12:03.609739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:03.609626 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" podStartSLOduration=1.609594932 podStartE2EDuration="1.609594932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:03.599326264 +0000 UTC m=+2382.039703460" watchObservedRunningTime="2026-10-30 06:12:03.609594932 +0000 UTC m=+2382.049972126" Oct 30 06:12:04.502197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:04.502157 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 06:12:04.502791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:04.502757 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 06:12:04.565022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:04.564988 6171 generic.go:356] "Generic (PLEG): container finished" podID="9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" containerID="c7e214427a2923aa8e61f6398f1aa571bc38b0995273e9a30ffd3cc9c7e2c8ea" exitCode=0 Oct 30 06:12:04.565235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:04.565075 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" event={"ID":"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02","Type":"ContainerDied","Data":"c7e214427a2923aa8e61f6398f1aa571bc38b0995273e9a30ffd3cc9c7e2c8ea"} Oct 30 06:12:05.114981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.114940 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:12:05.120813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.120787 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.136411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.135818 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:12:05.143499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.143469 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-cc33d3-openshift-default-dockercfg-7lll8\"" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178681 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178721 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178847 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178879 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178960 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fk9hd\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.178994 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.179059 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.179235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.179089 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.280907 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.280981 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281109 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281152 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281265 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281348 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281396 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281536 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fk9hd\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.285072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.281587 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.307068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.294424 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.307068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.295276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.307068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.295527 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.307068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.295758 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.307068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.295993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.311568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.310886 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.311568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.311476 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.321451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.321417 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.329702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.329664 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fk9hd\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.441200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.441101 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:05.503070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.502618 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" start-of-body= Oct 30 06:12:05.503070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.502680 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.41:15021/healthz/ready\": dial tcp 10.133.1.41:15021: connect: connection refused" Oct 30 06:12:05.818734 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:05.818104 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf52b07bb_95b7_4923_b212_b903da9f71b3.slice/crio-ed1729b35470b4de18795216a0b9d5d8ccfedc1a1c9ff7d75288969fe8e35c55 WatchSource:0}: Error finding container ed1729b35470b4de18795216a0b9d5d8ccfedc1a1c9ff7d75288969fe8e35c55: Status 404 returned error can't find the container with id ed1729b35470b4de18795216a0b9d5d8ccfedc1a1c9ff7d75288969fe8e35c55 Oct 30 06:12:05.818734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.818227 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:12:05.821979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.821944 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:05.822068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.822022 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:05.822106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.822089 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:05.830211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.830189 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:05.919917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.918741 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\") pod \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " Oct 30 06:12:05.919917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.918901 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\" (UniqueName: \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\") pod \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " Oct 30 06:12:05.919917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.918969 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\") pod \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\" (UID: \"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02\") " Oct 30 06:12:05.919917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.919227 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca" pod "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" (UID: "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:05.919917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.919881 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:05.922441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.922412 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx" pod "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" (UID: "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02"). InnerVolumeSpecName "kube-api-access-9btcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:05.924400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:05.924371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca" pod "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" (UID: "9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:06.021737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.021685 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9btcx\" (UniqueName: \"kubernetes.io/projected/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-kube-api-access-9btcx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:06.021737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.021729 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:06.493683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.493647 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:12:06.495116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.495087 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:12:06.608640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.608541 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" Oct 30 06:12:06.608640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.608598 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q" event={"ID":"9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02","Type":"ContainerDied","Data":"e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033"} Oct 30 06:12:06.609281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.608678 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e3c798fdb7af924a255fa07b8b8675e9396a025b042d22ef20e2a3b4b2a47033" Oct 30 06:12:06.613795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.613757 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" event={"ID":"f52b07bb-95b7-4923-b212-b903da9f71b3","Type":"ContainerStarted","Data":"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428"} Oct 30 06:12:06.613933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.613820 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" event={"ID":"f52b07bb-95b7-4923-b212-b903da9f71b3","Type":"ContainerStarted","Data":"ed1729b35470b4de18795216a0b9d5d8ccfedc1a1c9ff7d75288969fe8e35c55"} Oct 30 06:12:06.711066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:06.710982 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podStartSLOduration=1.7109652579999999 podStartE2EDuration="1.710965258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:06.68352531 +0000 UTC m=+2385.123902508" watchObservedRunningTime="2026-10-30 06:12:06.710965258 +0000 UTC m=+2385.151342448" Oct 30 06:12:07.445068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:07.442733 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:07.469129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:07.455215 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:07.469129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:07.455282 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:08.442333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:08.442302 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:08.442855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:08.442819 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:08.705121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:08.680091 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:12:09.441498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:09.441452 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:09.441828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:09.441512 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:10.441445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:10.441410 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:10.441926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:10.441532 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:11.442266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:11.442038 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:11.442689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:11.442302 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:12.441477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:12.441445 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" start-of-body= Oct 30 06:12:12.441663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:12.441497 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.63:15021/healthz/ready\": dial tcp 10.133.1.63:15021: connect: connection refused" Oct 30 06:12:13.075031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.074992 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp"] Oct 30 06:12:13.076240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.076219 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" containerName="revoke-keys" Oct 30 06:12:13.076240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.076239 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" containerName="revoke-keys" Oct 30 06:12:13.076464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.076423 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" containerName="revoke-keys" Oct 30 06:12:13.082020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.081991 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.092513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.092483 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp"] Oct 30 06:12:13.108357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.102438 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:13.117923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.117294 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:12:13.123436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.122371 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.123436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.122543 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s6hj\" (UniqueName: \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.123436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.122646 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.223742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.223700 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s6hj\" (UniqueName: \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.223957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.223788 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.223957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.223892 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.224569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.224543 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.226887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.226861 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.251524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.251492 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s6hj\" (UniqueName: \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\") pod \"maas-api-revoke-sub-ea494249-qbvcp\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.427121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.425374 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:13.456081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.454121 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:12:13.527137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.525364 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:13.634078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.623281 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-db0540\"" Oct 30 06:12:13.634078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.626222 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-db0540-dockercfg-hz2qv\"" Oct 30 06:12:13.731080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.712033 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:13.731080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.714160 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:13.731080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.728165 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:12:13.756067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.755618 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bkps\" (UniqueName: \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:13.756067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.755764 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:13.792154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.792117 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:12:13.802077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.802020 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:12:13.872087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.869584 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bkps\" (UniqueName: \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:13.872087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:13.869725 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:14.002339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.002268 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:14.037766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.037719 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bkps\" (UniqueName: \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\") pod \"payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:14.056464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.056406 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:12:14.065196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.063349 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.066519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.066498 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8"] Oct 30 06:12:14.073901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.073881 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.076379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.075991 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:12:14.085455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.084752 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:14.085455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.085266 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8"] Oct 30 06:12:14.094811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.094784 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-6b5b1e-openshift-default-dockercfg-n2kmp\"" Oct 30 06:12:14.178586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178557 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.178754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178619 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.178754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178730 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.178855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178812 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.178907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178880 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.178966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.178933 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp2pv\" (UniqueName: \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.179203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179178 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.179326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179306 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.179395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179354 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs72s\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.179455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179440 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.179573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179559 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.179645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.179607 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.281619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.281523 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.281799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.281678 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.281799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.281725 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.281991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.281879 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282354 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282421 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282460 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282524 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282527 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282589 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282618 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp2pv\" (UniqueName: \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282637 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282663 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282687 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs72s\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.283018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282745 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.283018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.282754 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.283359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.283308 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.284978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.284950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.285091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.284988 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.285465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.285448 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.324978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.324941 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp"] Oct 30 06:12:14.325771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.325746 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp2pv\" (UniqueName: \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\") pod \"maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.325908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.325869 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs72s\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.327514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.327491 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\") pod \"e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.329185 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:14.329144 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafe624e2_2de8_4819_981d_cb71566d1d24.slice/crio-f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc WatchSource:0}: Error finding container f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc: Status 404 returned error can't find the container with id f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc Oct 30 06:12:14.392215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.392182 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:14.399607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.399583 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:14.408518 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:14.408493 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0144da91_e849_44e2_8ac9_c8c6ad5033a5.slice/crio-a44f9672331853b33e752a3aacd6684ff9b76c627b3c8b0bd6935a88a9921fc6 WatchSource:0}: Error finding container a44f9672331853b33e752a3aacd6684ff9b76c627b3c8b0bd6935a88a9921fc6: Status 404 returned error can't find the container with id a44f9672331853b33e752a3aacd6684ff9b76c627b3c8b0bd6935a88a9921fc6 Oct 30 06:12:14.408621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.408597 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:12:14.756417 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:14.756383 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bd0af8c_fd93_4025_9885_f08673477722.slice/crio-1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e WatchSource:0}: Error finding container 1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e: Status 404 returned error can't find the container with id 1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e Oct 30 06:12:14.757432 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:14.757411 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0708855f_2a72_4f54_8ee5_2df4c073be69.slice/crio-2d1de47438aa9bc50a89af5fc2c469e3525f56e136328033b3b4b9af8413af38 WatchSource:0}: Error finding container 2d1de47438aa9bc50a89af5fc2c469e3525f56e136328033b3b4b9af8413af38: Status 404 returned error can't find the container with id 2d1de47438aa9bc50a89af5fc2c469e3525f56e136328033b3b4b9af8413af38 Oct 30 06:12:14.759121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.759101 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:12:14.760182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.760142 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:14.760310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.760223 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:14.760310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.760254 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:14.765403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:14.765379 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8"] Oct 30 06:12:15.077890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.077787 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" event={"ID":"0708855f-2a72-4f54-8ee5-2df4c073be69","Type":"ContainerStarted","Data":"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f"} Oct 30 06:12:15.077890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.077875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" event={"ID":"0708855f-2a72-4f54-8ee5-2df4c073be69","Type":"ContainerStarted","Data":"2d1de47438aa9bc50a89af5fc2c469e3525f56e136328033b3b4b9af8413af38"} Oct 30 06:12:15.079738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.079714 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" event={"ID":"0bd0af8c-fd93-4025-9885-f08673477722","Type":"ContainerStarted","Data":"1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e"} Oct 30 06:12:15.081839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.081811 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" event={"ID":"afe624e2-2de8-4819-981d-cb71566d1d24","Type":"ContainerStarted","Data":"7c2ff3a9bf700081540154d4277a23421e82e7d40ec213e79735b6dd3113c14f"} Oct 30 06:12:15.081960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.081867 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" event={"ID":"afe624e2-2de8-4819-981d-cb71566d1d24","Type":"ContainerStarted","Data":"f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc"} Oct 30 06:12:15.084255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.084231 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" event={"ID":"0144da91-e849-44e2-8ac9-c8c6ad5033a5","Type":"ContainerStarted","Data":"4d16df31bbf20e358f4725be7b7858584cda9a71b05e84dfb37ba206c875acc2"} Oct 30 06:12:15.084354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.084289 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" event={"ID":"0144da91-e849-44e2-8ac9-c8c6ad5033a5","Type":"ContainerStarted","Data":"a44f9672331853b33e752a3aacd6684ff9b76c627b3c8b0bd6935a88a9921fc6"} Oct 30 06:12:15.084436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.084410 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:15.133297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.133248 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podStartSLOduration=2.133234249 podStartE2EDuration="2.133234249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:15.127229118 +0000 UTC m=+2393.567606335" watchObservedRunningTime="2026-10-30 06:12:15.133234249 +0000 UTC m=+2393.573611441" Oct 30 06:12:15.244089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.244002 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" podStartSLOduration=2.243981754 podStartE2EDuration="2.243981754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:15.237539173 +0000 UTC m=+2393.677916361" watchObservedRunningTime="2026-10-30 06:12:15.243981754 +0000 UTC m=+2393.684358948" Oct 30 06:12:15.244582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.244544 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" podStartSLOduration=2.244535487 podStartE2EDuration="2.244535487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:15.190434393 +0000 UTC m=+2393.630811607" watchObservedRunningTime="2026-10-30 06:12:15.244535487 +0000 UTC m=+2393.684912682" Oct 30 06:12:15.392838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.392752 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:15.393921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.393882 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:15.393985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:15.393955 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:16.117828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.117789 6171 generic.go:356] "Generic (PLEG): container finished" podID="afe624e2-2de8-4819-981d-cb71566d1d24" containerID="7c2ff3a9bf700081540154d4277a23421e82e7d40ec213e79735b6dd3113c14f" exitCode=0 Oct 30 06:12:16.118339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.117872 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" event={"ID":"afe624e2-2de8-4819-981d-cb71566d1d24","Type":"ContainerDied","Data":"7c2ff3a9bf700081540154d4277a23421e82e7d40ec213e79735b6dd3113c14f"} Oct 30 06:12:16.120011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.119986 6171 generic.go:356] "Generic (PLEG): container finished" podID="0bd0af8c-fd93-4025-9885-f08673477722" containerID="4c533104d3a5594dcd241c8c6ebef67043e62c23cf602352246e22d581900219" exitCode=0 Oct 30 06:12:16.120189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.120077 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" event={"ID":"0bd0af8c-fd93-4025-9885-f08673477722","Type":"ContainerDied","Data":"4c533104d3a5594dcd241c8c6ebef67043e62c23cf602352246e22d581900219"} Oct 30 06:12:16.404129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.403756 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:16.404129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:16.403818 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:17.510357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.484537 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:17.510357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.484596 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:17.639693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.639665 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:17.725375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.725347 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:17.737465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.737438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\" (UniqueName: \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\") pod \"0bd0af8c-fd93-4025-9885-f08673477722\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " Oct 30 06:12:17.737663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.737646 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\") pod \"0bd0af8c-fd93-4025-9885-f08673477722\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " Oct 30 06:12:17.737748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.737733 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\") pod \"0bd0af8c-fd93-4025-9885-f08673477722\" (UID: \"0bd0af8c-fd93-4025-9885-f08673477722\") " Oct 30 06:12:17.738218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.738191 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca" pod "0bd0af8c-fd93-4025-9885-f08673477722" (UID: "0bd0af8c-fd93-4025-9885-f08673477722"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:17.738450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.738432 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:17.739850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.739828 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv" pod "0bd0af8c-fd93-4025-9885-f08673477722" (UID: "0bd0af8c-fd93-4025-9885-f08673477722"). InnerVolumeSpecName "kube-api-access-kp2pv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:17.739944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.739863 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca" pod "0bd0af8c-fd93-4025-9885-f08673477722" (UID: "0bd0af8c-fd93-4025-9885-f08673477722"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:17.862121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.840654 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\") pod \"afe624e2-2de8-4819-981d-cb71566d1d24\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " Oct 30 06:12:17.881068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.862772 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\" (UniqueName: \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\") pod \"afe624e2-2de8-4819-981d-cb71566d1d24\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " Oct 30 06:12:17.881442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.881413 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\") pod \"afe624e2-2de8-4819-981d-cb71566d1d24\" (UID: \"afe624e2-2de8-4819-981d-cb71566d1d24\") " Oct 30 06:12:17.907827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.907665 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kp2pv\" (UniqueName: \"kubernetes.io/projected/0bd0af8c-fd93-4025-9885-f08673477722-kube-api-access-kp2pv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:17.907827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.907712 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0bd0af8c-fd93-4025-9885-f08673477722-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:17.930354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:17.930272 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca" pod "afe624e2-2de8-4819-981d-cb71566d1d24" (UID: "afe624e2-2de8-4819-981d-cb71566d1d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:18.039140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.013258 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj" pod "afe624e2-2de8-4819-981d-cb71566d1d24" (UID: "afe624e2-2de8-4819-981d-cb71566d1d24"). InnerVolumeSpecName "kube-api-access-4s6hj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:18.039140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.016003 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s6hj\" (UniqueName: \"kubernetes.io/projected/afe624e2-2de8-4819-981d-cb71566d1d24-kube-api-access-4s6hj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:18.039140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.016036 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:18.039140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.016749 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca" pod "afe624e2-2de8-4819-981d-cb71566d1d24" (UID: "afe624e2-2de8-4819-981d-cb71566d1d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:18.117725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.117638 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/afe624e2-2de8-4819-981d-cb71566d1d24-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:18.140037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.140005 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" Oct 30 06:12:18.140244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.140029 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8" event={"ID":"0bd0af8c-fd93-4025-9885-f08673477722","Type":"ContainerDied","Data":"1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e"} Oct 30 06:12:18.140244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.140117 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1d5e38df1088f0e40c7e6fdcc7f2ad13d08880d21b92fe25bda2071b9b23423e" Oct 30 06:12:18.141821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.141801 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" Oct 30 06:12:18.141951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.141848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp" event={"ID":"afe624e2-2de8-4819-981d-cb71566d1d24","Type":"ContainerDied","Data":"f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc"} Oct 30 06:12:18.141951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.141892 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f35e4cff38c2ee1d17253a6cf1ab00568824ed430d3b7f21eaddb32cbaee3adc" Oct 30 06:12:18.405484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.405405 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:18.405484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:18.405460 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:19.402460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:19.402414 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:19.404443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:19.404408 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:20.405073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:20.399421 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:20.405073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:20.399484 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:21.392998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:21.392961 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" start-of-body= Oct 30 06:12:21.393240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:21.393026 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.66:15021/healthz/ready\": dial tcp 10.133.1.66:15021: connect: connection refused" Oct 30 06:12:22.398914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:22.398882 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:22.399423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:22.399404 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:22.400234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:22.400217 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:12:23.550854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.550817 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w"] Oct 30 06:12:23.551668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551653 6171 cpu_manager.go:414] "containerMap: removing container" podUID="afe624e2-2de8-4819-981d-cb71566d1d24" containerName="revoke-keys" Oct 30 06:12:23.551721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551670 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afe624e2-2de8-4819-981d-cb71566d1d24" containerName="revoke-keys" Oct 30 06:12:23.551721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551704 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0bd0af8c-fd93-4025-9885-f08673477722" containerName="revoke-keys" Oct 30 06:12:23.551721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551709 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bd0af8c-fd93-4025-9885-f08673477722" containerName="revoke-keys" Oct 30 06:12:23.551849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551839 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="afe624e2-2de8-4819-981d-cb71566d1d24" containerName="revoke-keys" Oct 30 06:12:23.551895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.551857 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bd0af8c-fd93-4025-9885-f08673477722" containerName="revoke-keys" Oct 30 06:12:23.557009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.556990 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.575722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.575697 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w"] Oct 30 06:12:23.583616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.583592 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:23.583788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.583682 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:12:23.627286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.627253 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdlhv\" (UniqueName: \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.627453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.627430 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.627676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.627661 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.729333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.729303 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdlhv\" (UniqueName: \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.729516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.729405 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.729585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.729534 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.730060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.730020 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.732134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.732111 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.733895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.733852 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerName="main" containerID="cri-o://c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691" gracePeriod=45 Oct 30 06:12:23.754791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.754767 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdlhv\" (UniqueName: \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\") pod \"maas-api-revoke-sub-43dfbf61-mvd4w\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:23.872850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:23.872818 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:24.041070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.040936 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:12:24.137124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137003 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\" (UniqueName: \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.137286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137247 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.137350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137282 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.137467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\" (UniqueName: \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.137860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137601 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:24.137860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137612 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.137860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.137819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:24.139062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.138207 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\") pod \"52dc24b0-2ccc-4e57-aa9d-289477eff462\" (UID: \"52dc24b0-2ccc-4e57-aa9d-289477eff462\") " Oct 30 06:12:24.139062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.138486 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:24.139062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.138975 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.139062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.138997 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.139062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.139011 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.139778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.139754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:24.139960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.139943 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:24.140141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.140122 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt" pod "52dc24b0-2ccc-4e57-aa9d-289477eff462" (UID: "52dc24b0-2ccc-4e57-aa9d-289477eff462"). InnerVolumeSpecName "kube-api-access-qd5gt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:24.214626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.214590 6171 generic.go:356] "Generic (PLEG): container finished" podID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerID="c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691" exitCode=0 Oct 30 06:12:24.214799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.214635 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" Oct 30 06:12:24.214799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.214670 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" event={"ID":"52dc24b0-2ccc-4e57-aa9d-289477eff462","Type":"ContainerDied","Data":"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691"} Oct 30 06:12:24.214799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.214715 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" event={"ID":"52dc24b0-2ccc-4e57-aa9d-289477eff462","Type":"ContainerDied","Data":"d134a4b2d221033ea5cbb9b20098fc0002ce02a73eebe59314ca017234521339"} Oct 30 06:12:24.214799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.214733 6171 scope.go:122] "RemoveContainer" containerID="c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691" Oct 30 06:12:24.228408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.228383 6171 scope.go:122] "RemoveContainer" containerID="c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691" Oct 30 06:12:24.228752 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:24.228727 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691\": container with ID starting with c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691 not found: ID does not exist" containerID="c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691" Oct 30 06:12:24.228872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.228759 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691"} err="failed to get container status \"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691\": rpc error: code = NotFound desc = could not find container \"c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691\": container with ID starting with c3695b829402616d4fac12e65a69ba1b6c6664f459ae0cd857e0a25622aec691 not found: ID does not exist" Oct 30 06:12:24.240343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.240315 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52dc24b0-2ccc-4e57-aa9d-289477eff462-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.240343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.240344 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qd5gt\" (UniqueName: \"kubernetes.io/projected/52dc24b0-2ccc-4e57-aa9d-289477eff462-kube-api-access-qd5gt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.240486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.240354 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52dc24b0-2ccc-4e57-aa9d-289477eff462-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:24.253016 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:24.252985 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0acc2e5b_aeb1_4d71_8989_71aa4a2600a5.slice/crio-3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b WatchSource:0}: Error finding container 3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b: Status 404 returned error can't find the container with id 3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b Oct 30 06:12:24.254958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.254935 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w"] Oct 30 06:12:24.303181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.303153 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:12:24.309032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.309007 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh"] Oct 30 06:12:24.432730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.432635 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:12:24.977621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:24.974188 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-03d349/auth-test-model-03d349-kserve-5f5bff785c-ksqlh" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerName="main" probeResult="failure" output="Get \"https://10.133.1.30:8000/ready\": context deadline exceeded" Oct 30 06:12:25.228730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:25.228642 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" event={"ID":"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5","Type":"ContainerStarted","Data":"0d44d8f8d209ee0e525c86c37874873c910d65dccfd11dd1fe11c2c3a34978f1"} Oct 30 06:12:25.228730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:25.228703 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" event={"ID":"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5","Type":"ContainerStarted","Data":"3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b"} Oct 30 06:12:25.279733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:25.279658 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" podStartSLOduration=2.279642623 podStartE2EDuration="2.279642623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:25.272328708 +0000 UTC m=+2403.712705903" watchObservedRunningTime="2026-10-30 06:12:25.279642623 +0000 UTC m=+2403.720019816" Oct 30 06:12:25.675139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:25.675035 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" path="/var/lib/kubelet/pods/52dc24b0-2ccc-4e57-aa9d-289477eff462/volumes" Oct 30 06:12:26.123383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:26.123338 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:12:26.257685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:26.257652 6171 generic.go:356] "Generic (PLEG): container finished" podID="0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" containerID="0d44d8f8d209ee0e525c86c37874873c910d65dccfd11dd1fe11c2c3a34978f1" exitCode=0 Oct 30 06:12:26.257892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:26.257736 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" event={"ID":"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5","Type":"ContainerDied","Data":"0d44d8f8d209ee0e525c86c37874873c910d65dccfd11dd1fe11c2c3a34978f1"} Oct 30 06:12:26.382574 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:26.382480 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:12:27.438767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.438743 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:27.530977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.530889 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\") pod \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " Oct 30 06:12:27.530977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.530968 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\" (UniqueName: \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\") pod \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " Oct 30 06:12:27.531362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.531339 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\") pod \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\" (UID: \"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5\") " Oct 30 06:12:27.532377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.532350 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca" pod "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" (UID: "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:27.533844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.533810 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca" pod "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" (UID: "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:27.534088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.534029 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv" pod "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" (UID: "0acc2e5b-aeb1-4d71-8989-71aa4a2600a5"). InnerVolumeSpecName "kube-api-access-xdlhv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:27.633552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.633519 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:27.633552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.633551 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdlhv\" (UniqueName: \"kubernetes.io/projected/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-kube-api-access-xdlhv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:27.633752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:27.633563 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:28.219315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:28.219281 6171 scope.go:122] "RemoveContainer" containerID="d7988cbced013c441840822c2e24cc8d0915769405975745587d4c9c647a7069" Oct 30 06:12:28.279078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:28.279054 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" Oct 30 06:12:28.279201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:28.279095 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w" event={"ID":"0acc2e5b-aeb1-4d71-8989-71aa4a2600a5","Type":"ContainerDied","Data":"3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b"} Oct 30 06:12:28.279201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:28.279150 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ab6aff1a8f054bfc92275800acb1d7870ff3d5f717353143db4df9b3dbbc86b" Oct 30 06:12:29.025473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.025434 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" podUID="ca09a4e2-2323-460e-aea3-40769aab5481" containerName="main" containerID="cri-o://4e925edb48a62638a9d477ae1bd6fe574fa674c0b61a759b9e6f364a56cdc33b" gracePeriod=45 Oct 30 06:12:29.364156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.364117 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:12:29.365550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.365522 6171 generic.go:356] "Generic (PLEG): container finished" podID="ca09a4e2-2323-460e-aea3-40769aab5481" containerID="4e925edb48a62638a9d477ae1bd6fe574fa674c0b61a759b9e6f364a56cdc33b" exitCode=0 Oct 30 06:12:29.365689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.365596 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" event={"ID":"ca09a4e2-2323-460e-aea3-40769aab5481","Type":"ContainerDied","Data":"4e925edb48a62638a9d477ae1bd6fe574fa674c0b61a759b9e6f364a56cdc33b"} Oct 30 06:12:29.365689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.365641 6171 scope.go:122] "RemoveContainer" containerID="4e925edb48a62638a9d477ae1bd6fe574fa674c0b61a759b9e6f364a56cdc33b" Oct 30 06:12:29.436829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.436130 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" containerID="cri-o://af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db" gracePeriod=25 Oct 30 06:12:29.464417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464388 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\" (UniqueName: \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464450 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464615 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464701 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464710 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:29.464756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464864 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\") pod \"ca09a4e2-2323-460e-aea3-40769aab5481\" (UID: \"ca09a4e2-2323-460e-aea3-40769aab5481\") " Oct 30 06:12:29.464960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.464926 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:29.465079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.465005 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:29.465514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.465459 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:29.465514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.465479 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:29.465514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.465489 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:29.466800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.466776 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:29.466800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.466795 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "kube-api-access-2mwzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:29.467191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.467175 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs" pod "ca09a4e2-2323-460e-aea3-40769aab5481" (UID: "ca09a4e2-2323-460e-aea3-40769aab5481"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:29.567573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.567511 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca09a4e2-2323-460e-aea3-40769aab5481-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:29.567573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.567535 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca09a4e2-2323-460e-aea3-40769aab5481-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:29.567573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:29.567545 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mwzz\" (UniqueName: \"kubernetes.io/projected/ca09a4e2-2323-460e-aea3-40769aab5481-kube-api-access-2mwzz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:30.375763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:30.375724 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" Oct 30 06:12:30.375763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:30.375743 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t" event={"ID":"ca09a4e2-2323-460e-aea3-40769aab5481","Type":"ContainerDied","Data":"883fbb34fd8f39aca5af32c5ae975537887ac3133e5caa50d5095d187b616944"} Oct 30 06:12:30.438335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:30.438295 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:12:30.452391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:30.452360 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-0d0152/test-model-0d0152-kserve-fd6967d47-7g72t"] Oct 30 06:12:31.674869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:31.674832 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca09a4e2-2323-460e-aea3-40769aab5481" path="/var/lib/kubelet/pods/ca09a4e2-2323-460e-aea3-40769aab5481/volumes" Oct 30 06:12:32.639665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:32.639608 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:12:34.463237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:34.463204 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 06:12:34.463705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:34.463431 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" containerID="cri-o://519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf" gracePeriod=30 Oct 30 06:12:36.034615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:36.034581 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg"] Oct 30 06:12:36.047517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:36.047495 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn"] Oct 30 06:12:36.052392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:36.052367 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88df6884-pqffg"] Oct 30 06:12:36.056594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:36.056570 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bb936e39-kgpvn"] Oct 30 06:12:37.691862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:37.691751 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108f5f78-40c4-4fe9-bbd3-7b249c998f23" path="/var/lib/kubelet/pods/108f5f78-40c4-4fe9-bbd3-7b249c998f23/volumes" Oct 30 06:12:37.693599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:37.693561 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf21affb-a65e-40cd-be69-9833a5a06a92" path="/var/lib/kubelet/pods/bf21affb-a65e-40cd-be69-9833a5a06a92/volumes" Oct 30 06:12:37.790076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:37.790032 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8"] Oct 30 06:12:37.803460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:37.803436 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d3e7e6-j6jz8"] Oct 30 06:12:39.430313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:39.430247 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:12:39.431060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:39.431010 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:12:39.720152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:39.720067 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bd0af8c-fd93-4025-9885-f08673477722" path="/var/lib/kubelet/pods/0bd0af8c-fd93-4025-9885-f08673477722/volumes" Oct 30 06:12:40.601300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.601262 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 06:12:40.668694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.667573 6171 generic.go:356] "Generic (PLEG): container finished" podID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerID="519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf" exitCode=0 Oct 30 06:12:40.668694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.668012 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerDied","Data":"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf"} Oct 30 06:12:40.668694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.668092 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" event={"ID":"7756216c-0f8a-4f1b-bc98-b31c0c561ac1","Type":"ContainerDied","Data":"ad080e3022222977d2bd593e19dde34c5101b591586c1896cd78776ad6b83c79"} Oct 30 06:12:40.668694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.668128 6171 scope.go:122] "RemoveContainer" containerID="519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf" Oct 30 06:12:40.668694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.668318 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4" Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680016 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680166 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680234 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680315 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680397 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.680729 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.681521 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.681596 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.681725 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.684604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.682388 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\") pod \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\" (UID: \"7756216c-0f8a-4f1b-bc98-b31c0c561ac1\") " Oct 30 06:12:40.687584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.687362 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:40.710133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.687803 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:40.710133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.698580 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:40.710133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.698604 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:40.710133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.698618 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:40.710133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.703127 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:40.712405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.711251 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:40.754068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.753506 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:40.975235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.951251 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:40.975235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.951292 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:40.975235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:40.951312 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:41.226335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.213122 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:12:41.226335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.215389 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:12:41.226335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.215512 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "kube-api-access-t94p8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:41.226335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.215566 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token" pod "7756216c-0f8a-4f1b-bc98-b31c0c561ac1" (UID: "7756216c-0f8a-4f1b-bc98-b31c0c561ac1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:41.226335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.217815 6171 scope.go:122] "RemoveContainer" containerID="56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7" Oct 30 06:12:41.246319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.246236 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:41.246319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.246273 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94p8\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-kube-api-access-t94p8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:41.246319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.246289 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7756216c-0f8a-4f1b-bc98-b31c0c561ac1-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:41.252576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.252248 6171 scope.go:122] "RemoveContainer" containerID="519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf" Oct 30 06:12:41.272092 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:41.270469 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf\": container with ID starting with 519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf not found: ID does not exist" containerID="519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf" Oct 30 06:12:41.272092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.270550 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf"} err="failed to get container status \"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf\": rpc error: code = NotFound desc = could not find container \"519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf\": container with ID starting with 519ce879aa8e31735ad4257c6e0bdee7691c59c351365b142897e245fa38dbdf not found: ID does not exist" Oct 30 06:12:41.272092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.270578 6171 scope.go:122] "RemoveContainer" containerID="56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7" Oct 30 06:12:41.277062 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:41.273724 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7\": container with ID starting with 56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7 not found: ID does not exist" containerID="56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7" Oct 30 06:12:41.277062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.273790 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7"} err="failed to get container status \"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7\": rpc error: code = NotFound desc = could not find container \"56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7\": container with ID starting with 56e9fc10209d7ca50adab6f3fd110cffbb0340858eb998d70a376e56595731c7 not found: ID does not exist" Oct 30 06:12:41.358213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.358174 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 06:12:41.368080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.368023 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-wglw4"] Oct 30 06:12:41.729125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:41.723619 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" path="/var/lib/kubelet/pods/7756216c-0f8a-4f1b-bc98-b31c0c561ac1/volumes" Oct 30 06:12:42.640528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:42.640495 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:12:45.365935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.365905 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:12:45.366604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.366574 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" containerID="cri-o://ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b" gracePeriod=30 Oct 30 06:12:45.921619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.921582 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k"] Oct 30 06:12:45.922456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922435 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ca09a4e2-2323-460e-aea3-40769aab5481" containerName="main" Oct 30 06:12:45.922456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922454 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca09a4e2-2323-460e-aea3-40769aab5481" containerName="main" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922473 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922478 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922491 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" containerName="revoke-keys" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922496 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" containerName="revoke-keys" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922515 6171 cpu_manager.go:414] "containerMap: removing container" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerName="main" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922519 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerName="main" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922666 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca09a4e2-2323-460e-aea3-40769aab5481" containerName="main" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922675 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922684 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" containerName="revoke-keys" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922693 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="52dc24b0-2ccc-4e57-aa9d-289477eff462" containerName="main" Oct 30 06:12:45.922698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.922701 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:45.928291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.928273 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:45.954441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:45.954408 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k"] Oct 30 06:12:46.006954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.006917 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.007223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.007096 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jtk7\" (UniqueName: \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.007223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.007184 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.019543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.019500 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:12:46.019691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.019649 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:12:46.109057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.109006 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.109250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.109119 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jtk7\" (UniqueName: \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.109250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.109191 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.109701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.109674 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.111805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.111782 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.134325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.134299 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jtk7\" (UniqueName: \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\") pod \"maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.260568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.260531 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:46.612481 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:46.612446 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1c64fc0_226e_49a1_a5cb_fc2fba2dedf8.slice/crio-1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd WatchSource:0}: Error finding container 1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd: Status 404 returned error can't find the container with id 1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd Oct 30 06:12:46.612969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.612725 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k"] Oct 30 06:12:46.818313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:46.818280 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" event={"ID":"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8","Type":"ContainerStarted","Data":"1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd"} Oct 30 06:12:47.828830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:47.828800 6171 generic.go:356] "Generic (PLEG): container finished" podID="b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" containerID="75b96b1b152b5c7d2893bdfcbfcf66c8be555239b63f80c5d069f76c65d2736a" exitCode=0 Oct 30 06:12:47.829307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:47.828848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" event={"ID":"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8","Type":"ContainerDied","Data":"75b96b1b152b5c7d2893bdfcbfcf66c8be555239b63f80c5d069f76c65d2736a"} Oct 30 06:12:49.137926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.137883 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:49.181742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.181707 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\") pod \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " Oct 30 06:12:49.181957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.181857 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\" (UniqueName: \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\") pod \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " Oct 30 06:12:49.182133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.182112 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\") pod \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\" (UID: \"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8\") " Oct 30 06:12:49.182521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.182133 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca" pod "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" (UID: "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:49.183133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.183109 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:49.185530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.184961 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7" pod "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" (UID: "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8"). InnerVolumeSpecName "kube-api-access-4jtk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:49.195300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.195275 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca" pod "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" (UID: "b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:12:49.284686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.284654 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:49.284686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.284686 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jtk7\" (UniqueName: \"kubernetes.io/projected/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8-kube-api-access-4jtk7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:49.788347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.788311 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:12:49.803521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.803497 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\" (UniqueName: \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\") pod \"8df40470-788d-4c78-85e5-32c2b75535b8\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " Oct 30 06:12:49.803668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.803562 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\") pod \"8df40470-788d-4c78-85e5-32c2b75535b8\" (UID: \"8df40470-788d-4c78-85e5-32c2b75535b8\") " Oct 30 06:12:49.803994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.803964 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume" pod "8df40470-788d-4c78-85e5-32c2b75535b8" (UID: "8df40470-788d-4c78-85e5-32c2b75535b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:49.804377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.804355 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8df40470-788d-4c78-85e5-32c2b75535b8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:49.805730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.805706 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8" pod "8df40470-788d-4c78-85e5-32c2b75535b8" (UID: "8df40470-788d-4c78-85e5-32c2b75535b8"). InnerVolumeSpecName "kube-api-access-m4zt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:49.857035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.856998 6171 generic.go:356] "Generic (PLEG): container finished" podID="8df40470-788d-4c78-85e5-32c2b75535b8" containerID="af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db" exitCode=0 Oct 30 06:12:49.857244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.857075 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" Oct 30 06:12:49.857244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.857083 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" event={"ID":"8df40470-788d-4c78-85e5-32c2b75535b8","Type":"ContainerDied","Data":"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db"} Oct 30 06:12:49.857244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.857216 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f" event={"ID":"8df40470-788d-4c78-85e5-32c2b75535b8","Type":"ContainerDied","Data":"b8c12a02f29654b0f1ced029e686bc648faeb296771789da6fed2f36df67b157"} Oct 30 06:12:49.857244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.857243 6171 scope.go:122] "RemoveContainer" containerID="af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db" Oct 30 06:12:49.859398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.859367 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" event={"ID":"b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8","Type":"ContainerDied","Data":"1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd"} Oct 30 06:12:49.859674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.859423 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1138a67db3189ad4ab86ade126a907749ad8614026ae36754ff17024fce6f9cd" Oct 30 06:12:49.859674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.859480 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k" Oct 30 06:12:49.874391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.874372 6171 scope.go:122] "RemoveContainer" containerID="af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db" Oct 30 06:12:49.874687 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:49.874663 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db\": container with ID starting with af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db not found: ID does not exist" containerID="af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db" Oct 30 06:12:49.874773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.874697 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db"} err="failed to get container status \"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db\": rpc error: code = NotFound desc = could not find container \"af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db\": container with ID starting with af94ddeb72a726c4eea595c2341b08126c51bd52993ba8c2387a29e1aadf85db not found: ID does not exist" Oct 30 06:12:49.906331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:49.906306 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4zt8\" (UniqueName: \"kubernetes.io/projected/8df40470-788d-4c78-85e5-32c2b75535b8-kube-api-access-m4zt8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:50.011099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.007096 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:12:50.045473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.042333 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d3e7e6-57cfbbcf-xht6f"] Oct 30 06:12:50.526890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.526849 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:12:50.528213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528187 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:50.528213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528213 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7756216c-0f8a-4f1b-bc98-b31c0c561ac1" containerName="istio-proxy" Oct 30 06:12:50.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528242 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" Oct 30 06:12:50.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528247 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" Oct 30 06:12:50.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528264 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" containerName="revoke-keys" Oct 30 06:12:50.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528268 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" containerName="revoke-keys" Oct 30 06:12:50.528407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528405 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" containerName="payload-pre-processing" Oct 30 06:12:50.528720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.528420 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" containerName="revoke-keys" Oct 30 06:12:50.532938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.532912 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.552822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.552745 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"kube-root-ca.crt\"" Oct 30 06:12:50.556233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.556204 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:12:50.562776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.562689 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"auth-test-model-d1bfce-kserve-self-signed-certs\"" Oct 30 06:12:50.562904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.562808 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"default-dockercfg-xgkjl\"" Oct 30 06:12:50.562904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.562882 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w5-d1bfce\"/\"openshift-service-ca.crt\"" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621377 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621459 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621616 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwhl5\" (UniqueName: \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621647 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621737 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.624107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.621809 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.705396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.705375 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:12:50.723937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.723908 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724003 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724191 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwhl5\" (UniqueName: \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724238 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724355 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724404 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724442 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724624 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.724791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.724769 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.726827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.726800 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.726925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.726871 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.751772 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.751746 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwhl5\" (UniqueName: \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\") pod \"auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.825429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825391 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825511 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825570 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825638 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825709 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825767 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825817 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.825952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.825880 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.826117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.826004 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\") pod \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\" (UID: \"ae869cc6-d981-4fe6-86ca-36a4eabd6b95\") " Oct 30 06:12:50.827362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.827332 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.827565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.827541 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.827693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.827674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.827930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.827904 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:12:50.828812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.828784 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.830009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.829989 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "kube-api-access-cbbvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:50.830171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.830149 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:12:50.830554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.830538 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:12:50.830975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.830963 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo" pod "ae869cc6-d981-4fe6-86ca-36a4eabd6b95" (UID: "ae869cc6-d981-4fe6-86ca-36a4eabd6b95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:12:50.863611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.863573 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:50.894186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.893738 6171 generic.go:356] "Generic (PLEG): container finished" podID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerID="ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b" exitCode=0 Oct 30 06:12:50.894186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.893943 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerDied","Data":"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b"} Oct 30 06:12:50.894186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.894014 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" event={"ID":"ae869cc6-d981-4fe6-86ca-36a4eabd6b95","Type":"ContainerDied","Data":"7d1a71dd4eb74ea024f51524bf96f4f33ea59893fbe87d02e0a29c29b4396645"} Oct 30 06:12:50.894186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.894085 6171 scope.go:122] "RemoveContainer" containerID="ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b" Oct 30 06:12:50.896069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.894691 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975200 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975231 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbbvl\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-kube-api-access-cbbvl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975251 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975264 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975276 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975287 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975297 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975307 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.005092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:50.975318 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae869cc6-d981-4fe6-86ca-36a4eabd6b95-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:12:51.061450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.061370 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn"] Oct 30 06:12:51.070593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.070564 6171 scope.go:122] "RemoveContainer" containerID="70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2" Oct 30 06:12:51.075078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.075032 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86"] Oct 30 06:12:51.086566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.086533 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-37ae9cd8-zlt86"] Oct 30 06:12:51.093442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.092157 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b24fc33-j79bn"] Oct 30 06:12:51.095142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.095123 6171 scope.go:122] "RemoveContainer" containerID="ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b" Oct 30 06:12:51.095713 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:51.095688 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b\": container with ID starting with ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b not found: ID does not exist" containerID="ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b" Oct 30 06:12:51.095830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.095730 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b"} err="failed to get container status \"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b\": rpc error: code = NotFound desc = could not find container \"ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b\": container with ID starting with ba3390ca937332f09e4ce84aca17b51029f67b13f1f5948c28b4a3c5d1cd6a2b not found: ID does not exist" Oct 30 06:12:51.095830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.095756 6171 scope.go:122] "RemoveContainer" containerID="70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2" Oct 30 06:12:51.096363 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:51.096308 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2\": container with ID starting with 70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2 not found: ID does not exist" containerID="70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2" Oct 30 06:12:51.096363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.096350 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2"} err="failed to get container status \"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2\": rpc error: code = NotFound desc = could not find container \"70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2\": container with ID starting with 70eafbb35691b39170d43009a9512edbcc68a55afad521db2fee0be0e1d87fb2 not found: ID does not exist" Oct 30 06:12:51.119349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.119321 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:12:51.123974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.123950 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-d3e7e6-openshift-default-6c65f9d6ff-7fhz7"] Oct 30 06:12:51.383798 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:51.383766 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1f93a83_d39a_4070_8c39_94e94afe1d31.slice/crio-39c3b619f8676c2a5ccc3e97e3a2c6d56b7a8756deffa1df158c0ec3fc5aee96 WatchSource:0}: Error finding container 39c3b619f8676c2a5ccc3e97e3a2c6d56b7a8756deffa1df158c0ec3fc5aee96: Status 404 returned error can't find the container with id 39c3b619f8676c2a5ccc3e97e3a2c6d56b7a8756deffa1df158c0ec3fc5aee96 Oct 30 06:12:51.383973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.383951 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:12:51.681565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.681486 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="539e68d2-4ece-47e7-aed6-4e727d3173be" path="/var/lib/kubelet/pods/539e68d2-4ece-47e7-aed6-4e727d3173be/volumes" Oct 30 06:12:51.681991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.681935 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8df40470-788d-4c78-85e5-32c2b75535b8" path="/var/lib/kubelet/pods/8df40470-788d-4c78-85e5-32c2b75535b8/volumes" Oct 30 06:12:51.682473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.682456 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9187ee-2869-426d-9aa3-230ff2d45fee" path="/var/lib/kubelet/pods/9f9187ee-2869-426d-9aa3-230ff2d45fee/volumes" Oct 30 06:12:51.683228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.683209 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" path="/var/lib/kubelet/pods/ae869cc6-d981-4fe6-86ca-36a4eabd6b95/volumes" Oct 30 06:12:51.921475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.921444 6171 generic.go:356] "Generic (PLEG): container finished" podID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerID="beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176" exitCode=0 Oct 30 06:12:51.921668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.921519 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerDied","Data":"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176"} Oct 30 06:12:51.921993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.921971 6171 scope.go:122] "RemoveContainer" containerID="beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176" Oct 30 06:12:51.923560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.923533 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" event={"ID":"e1f93a83-d39a-4070-8c39-94e94afe1d31","Type":"ContainerStarted","Data":"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02"} Oct 30 06:12:51.923690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.923566 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" event={"ID":"e1f93a83-d39a-4070-8c39-94e94afe1d31","Type":"ContainerStarted","Data":"39c3b619f8676c2a5ccc3e97e3a2c6d56b7a8756deffa1df158c0ec3fc5aee96"} Oct 30 06:12:51.923818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.923786 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:51.923891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.923855 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:51.923891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:51.923885 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:12:52.118149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:52.118086 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" podStartSLOduration=2.118066738 podStartE2EDuration="2.118066738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:52.113549905 +0000 UTC m=+2430.553927098" watchObservedRunningTime="2026-10-30 06:12:52.118066738 +0000 UTC m=+2430.558443932" Oct 30 06:12:52.864308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:52.864279 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:12:52.935698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:52.935660 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerStarted","Data":"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f"} Oct 30 06:12:53.351123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.351084 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:12:53.351723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.351697 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:53.351850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.351756 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:53.855298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.854245 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:12:53.855488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.855318 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:12:53.855488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.855338 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:12:53.855488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.855398 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:12:53.855488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.855409 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:12:53.855645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.855609 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:12:53.860014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.859990 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.914395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.913355 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-f22a52\"/\"kube-root-ca.crt\"" Oct 30 06:12:53.914395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.914028 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w6-f22a52\"/\"openshift-service-ca.crt\"" Oct 30 06:12:53.914395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.914191 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-f22a52\"/\"default-dockercfg-kxrpj\"" Oct 30 06:12:53.914395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.914304 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w6-f22a52\"/\"auth-test-model-f22a52-kserve-self-signed-certs\"" Oct 30 06:12:53.918624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.918251 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:12:53.925634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.925596 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.925749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.925648 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.925749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.925696 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkmnk\" (UniqueName: \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.925845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.925812 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.926998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.925950 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:53.926998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:53.926069 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027143 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkmnk\" (UniqueName: \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027273 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027702 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027861 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.027892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027873 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.028128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.027922 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.028128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.028016 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.028260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.028240 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.030145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.030125 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.030221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.030150 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.052233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.052212 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkmnk\" (UniqueName: \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\") pod \"auth-test-model-f22a52-kserve-666964888f-tmr72\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.180133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.179338 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:54.352057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.351995 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:54.352266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.352090 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:54.638700 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:12:54.638667 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod42def66a_46ba_4886_abf5_244ec2ad4a38.slice/crio-5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3 WatchSource:0}: Error finding container 5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3: Status 404 returned error can't find the container with id 5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3 Oct 30 06:12:54.640956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.640413 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:12:54.963490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.963399 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" event={"ID":"42def66a-46ba-4886-abf5-244ec2ad4a38","Type":"ContainerStarted","Data":"bde3a8a9fffb7358d1205e2f01739c486d6dcc3b252272e5b0d64249a2b213e5"} Oct 30 06:12:54.963490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:54.963460 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" event={"ID":"42def66a-46ba-4886-abf5-244ec2ad4a38","Type":"ContainerStarted","Data":"5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3"} Oct 30 06:12:55.195071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:55.180322 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:12:55.351225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:55.351187 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:55.351423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:55.351238 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:56.351807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:56.351771 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:56.352310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:56.351834 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:56.429400 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:12:56.429357 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:12:57.351413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:57.351378 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:57.351703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:57.351676 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:58.351549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:58.351512 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" start-of-body= Oct 30 06:12:58.351974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:58.351576 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.56:15021/healthz/ready\": dial tcp 10.133.1.56:15021: connect: connection refused" Oct 30 06:12:59.356282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:59.356254 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:12:59.356757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:59.356728 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:12:59.357396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:59.357380 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:12:59.424107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:12:59.424067 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" podStartSLOduration=6.424053493 podStartE2EDuration="6.424053493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:55.049937947 +0000 UTC m=+2433.490315140" watchObservedRunningTime="2026-10-30 06:12:59.424053493 +0000 UTC m=+2437.864430683" Oct 30 06:13:00.864605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:00.864568 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:13:00.878202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:00.878179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:13:01.089018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.088992 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:13:01.944122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.940571 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:13:01.947107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.947069 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae869cc6-d981-4fe6-86ca-36a4eabd6b95" containerName="istio-proxy" Oct 30 06:13:01.952194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.952170 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:13:01.952497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.952478 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:01.970983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.970956 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"openshift-service-ca.crt\"" Oct 30 06:13:01.985314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.985287 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"kube-root-ca.crt\"" Oct 30 06:13:01.985463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.985400 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"auth-test-model-24db22-kserve-self-signed-certs\"" Oct 30 06:13:01.985719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:01.985700 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-24db22\"/\"default-dockercfg-ngbhd\"" Oct 30 06:13:02.046443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046409 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbm9c\" (UniqueName: \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.046597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046482 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.046597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046564 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.046684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046669 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.046849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046836 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.046898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.046883 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148230 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148286 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148376 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148640 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148767 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148752 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbm9c\" (UniqueName: \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.148883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148763 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.149021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.148990 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.149437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.149417 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.150920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.150898 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.151513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.151491 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.220127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.219921 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbm9c\" (UniqueName: \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\") pod \"auth-test-model-24db22-kserve-6bb4d9d85c-p9szq\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.277075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.276384 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:02.611239 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:02.611208 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4fe47436_0fbf_403d_8b56_9485a5572b27.slice/crio-59d61d438df05aaf0cf2c9a0ca66451f0415cfc3698551ea49bced8ece3decf9 WatchSource:0}: Error finding container 59d61d438df05aaf0cf2c9a0ca66451f0415cfc3698551ea49bced8ece3decf9: Status 404 returned error can't find the container with id 59d61d438df05aaf0cf2c9a0ca66451f0415cfc3698551ea49bced8ece3decf9 Oct 30 06:13:02.615430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:02.615400 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:13:03.100281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:03.100239 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" event={"ID":"4fe47436-0fbf-403d-8b56-9485a5572b27","Type":"ContainerStarted","Data":"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229"} Oct 30 06:13:03.100762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:03.100292 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" event={"ID":"4fe47436-0fbf-403d-8b56-9485a5572b27","Type":"ContainerStarted","Data":"59d61d438df05aaf0cf2c9a0ca66451f0415cfc3698551ea49bced8ece3decf9"} Oct 30 06:13:03.150034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:03.149980 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" podStartSLOduration=2.149965109 podStartE2EDuration="2.149965109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:03.145947499 +0000 UTC m=+2441.586324683" watchObservedRunningTime="2026-10-30 06:13:03.149965109 +0000 UTC m=+2441.590342299" Oct 30 06:13:03.284148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:03.276503 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:04.183415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.183378 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:13:04.197121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.197090 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:13:04.561073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.561023 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:13:04.566150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.566131 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.584246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.584225 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-c73fcb-openshift-default-dockercfg-rtknk\"" Oct 30 06:13:04.605142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.605109 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:13:04.701158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701108 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcpvp\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701172 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701309 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701386 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701451 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701483 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701611 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701758 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.701858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.701843 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.803888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.803852 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.803917 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804056 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcpvp\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804094 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804117 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804259 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804309 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804347 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804435 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804448 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804510 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804603 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804642 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.804968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.804947 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.806696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.806674 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.807023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.807006 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.833835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.833780 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.834015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.833990 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcpvp\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\") pod \"e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:04.883432 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:04.883407 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:05.129793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:05.129716 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:13:05.225628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:05.225599 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:13:05.227095 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:05.227065 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda2ddb7_892d_4784_8936_d5bfcafc98cf.slice/crio-b1df7fe5d39ab85f005ac996691c60fcbfcb153d4c68ab8a69223505f08b8758 WatchSource:0}: Error finding container b1df7fe5d39ab85f005ac996691c60fcbfcb153d4c68ab8a69223505f08b8758: Status 404 returned error can't find the container with id b1df7fe5d39ab85f005ac996691c60fcbfcb153d4c68ab8a69223505f08b8758 Oct 30 06:13:05.230440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:05.230404 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:05.230520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:05.230491 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:05.230557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:05.230534 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:06.129970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.129929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" event={"ID":"bda2ddb7-892d-4784-8936-d5bfcafc98cf","Type":"ContainerStarted","Data":"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f"} Oct 30 06:13:06.130205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.129987 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" event={"ID":"bda2ddb7-892d-4784-8936-d5bfcafc98cf","Type":"ContainerStarted","Data":"b1df7fe5d39ab85f005ac996691c60fcbfcb153d4c68ab8a69223505f08b8758"} Oct 30 06:13:06.208108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.205138 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podStartSLOduration=2.205116124 podStartE2EDuration="2.205116124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:06.192704017 +0000 UTC m=+2444.633081211" watchObservedRunningTime="2026-10-30 06:13:06.205116124 +0000 UTC m=+2444.645493317" Oct 30 06:13:06.884020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.883988 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:06.897068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.892202 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:13:06.897068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:06.892290 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" Oct 30 06:13:07.896474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:07.896432 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:13:07.910979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:07.910940 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" Oct 30 06:13:08.889421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:08.889384 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:13:08.889724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:08.889692 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" Oct 30 06:13:09.888368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:09.888325 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:13:09.889012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:09.888979 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" Oct 30 06:13:10.888346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:10.888302 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" start-of-body= Oct 30 06:13:10.888633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:10.888603 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.73:15021/healthz/ready\": dial tcp 10.133.1.73:15021: connect: connection refused" Oct 30 06:13:12.017254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:12.017217 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:12.017780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:12.017613 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:12.018590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:12.018562 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:13:12.276696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:12.276609 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:12.295316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:12.295292 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:13.215172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:13.215126 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:13:21.075707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:21.075667 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k"] Oct 30 06:13:21.085430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:21.085399 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-0d0152-pz92k"] Oct 30 06:13:21.678313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:21.678272 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8" path="/var/lib/kubelet/pods/b1c64fc0-226e-49a1-a5cb-fc2fba2dedf8/volumes" Oct 30 06:13:26.382269 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:13:26.382230 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:13:27.648780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:27.648740 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:13:28.191375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:28.191340 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:13:28.283431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:28.283389 6171 scope.go:122] "RemoveContainer" containerID="2c09cec5d77940711f370a098a6b0f863c5c925276c536df336c4584d1a2da65" Oct 30 06:13:28.300920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:28.300895 6171 scope.go:122] "RemoveContainer" containerID="1684dcaab9230b43925d2f95b85fd6d7f9d8bf403b9cfa25bd0e93bc438e9d2a" Oct 30 06:13:30.080413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:30.080374 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:13:30.080801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:30.080630 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" containerID="cri-o://1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b" gracePeriod=30 Oct 30 06:13:33.192580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.192514 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" podUID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" containerName="payload-processing" containerID="cri-o://4d16df31bbf20e358f4725be7b7858584cda9a71b05e84dfb37ba206c875acc2" gracePeriod=25 Oct 30 06:13:33.640318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.640282 6171 generic.go:356] "Generic (PLEG): container finished" podID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" containerID="4d16df31bbf20e358f4725be7b7858584cda9a71b05e84dfb37ba206c875acc2" exitCode=0 Oct 30 06:13:33.640908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.640883 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" event={"ID":"0144da91-e849-44e2-8ac9-c8c6ad5033a5","Type":"ContainerDied","Data":"4d16df31bbf20e358f4725be7b7858584cda9a71b05e84dfb37ba206c875acc2"} Oct 30 06:13:33.657836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.657816 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:13:33.746851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.746816 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\" (UniqueName: \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\") pod \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " Oct 30 06:13:33.747159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.747137 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\") pod \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\" (UID: \"0144da91-e849-44e2-8ac9-c8c6ad5033a5\") " Oct 30 06:13:33.747591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.747563 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume" pod "0144da91-e849-44e2-8ac9-c8c6ad5033a5" (UID: "0144da91-e849-44e2-8ac9-c8c6ad5033a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:33.747895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.747877 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0144da91-e849-44e2-8ac9-c8c6ad5033a5-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:33.749785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.749754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps" pod "0144da91-e849-44e2-8ac9-c8c6ad5033a5" (UID: "0144da91-e849-44e2-8ac9-c8c6ad5033a5"). InnerVolumeSpecName "kube-api-access-6bkps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:33.849617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:33.849586 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bkps\" (UniqueName: \"kubernetes.io/projected/0144da91-e849-44e2-8ac9-c8c6ad5033a5-kube-api-access-6bkps\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:34.651858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.651827 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" Oct 30 06:13:34.652297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.651865 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp" event={"ID":"0144da91-e849-44e2-8ac9-c8c6ad5033a5","Type":"ContainerDied","Data":"a44f9672331853b33e752a3aacd6684ff9b76c627b3c8b0bd6935a88a9921fc6"} Oct 30 06:13:34.652297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.651958 6171 scope.go:122] "RemoveContainer" containerID="4d16df31bbf20e358f4725be7b7858584cda9a71b05e84dfb37ba206c875acc2" Oct 30 06:13:34.716073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.716008 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:13:34.728204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.728134 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-db0540-c8f7b8fd7-6f7lp"] Oct 30 06:13:34.729674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:34.729647 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:13:35.388919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.388895 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494699 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494753 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494793 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494883 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494931 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.494985 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.495158 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.495212 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.495419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.495380 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\") pod \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\" (UID: \"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d\") " Oct 30 06:13:35.500533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.499630 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.500533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.500006 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.500533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.500221 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.504416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.503282 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.504676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.504658 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.504792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.504777 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.504897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.504885 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.523879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.521334 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:35.523879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.523471 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:35.523879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.523696 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "kube-api-access-pkg5x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:35.523879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.523772 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:35.527613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.526255 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo" pod "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" (UID: "8a9794fc-a2c6-4710-a6d9-643fcaaafb5d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:13:35.607032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607002 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkg5x\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-kube-api-access-pkg5x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.607032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607031 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.607240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607058 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.607240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607070 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.607240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607080 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.607240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.607088 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:35.664325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.664246 6171 generic.go:356] "Generic (PLEG): container finished" podID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerID="1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b" exitCode=0 Oct 30 06:13:35.664785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.664376 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" Oct 30 06:13:35.676873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.676849 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" path="/var/lib/kubelet/pods/0144da91-e849-44e2-8ac9-c8c6ad5033a5/volumes" Oct 30 06:13:35.677512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.677489 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerDied","Data":"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b"} Oct 30 06:13:35.677598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.677520 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7" event={"ID":"8a9794fc-a2c6-4710-a6d9-643fcaaafb5d","Type":"ContainerDied","Data":"75d978d3d387df4a3272b6e0db1b99742a14a723a504da2e346cc651d39fac6c"} Oct 30 06:13:35.677598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.677546 6171 scope.go:122] "RemoveContainer" containerID="1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b" Oct 30 06:13:35.691169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.691149 6171 scope.go:122] "RemoveContainer" containerID="3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56" Oct 30 06:13:35.704778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.704759 6171 scope.go:122] "RemoveContainer" containerID="1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b" Oct 30 06:13:35.705088 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:13:35.705031 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b\": container with ID starting with 1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b not found: ID does not exist" containerID="1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b" Oct 30 06:13:35.705170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.705098 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b"} err="failed to get container status \"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b\": rpc error: code = NotFound desc = could not find container \"1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b\": container with ID starting with 1e06fdbb1928a2952a5fca0011953458831ffa3ecac4d379ee93ee4d2229869b not found: ID does not exist" Oct 30 06:13:35.705170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.705125 6171 scope.go:122] "RemoveContainer" containerID="3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56" Oct 30 06:13:35.705375 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:13:35.705354 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56\": container with ID starting with 3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56 not found: ID does not exist" containerID="3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56" Oct 30 06:13:35.705416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.705383 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56"} err="failed to get container status \"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56\": rpc error: code = NotFound desc = could not find container \"3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56\": container with ID starting with 3550ee0c666dff50742dca8aa551e7e7294c02b0501c104c5db64a13a43dba56 not found: ID does not exist" Oct 30 06:13:35.781905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.781875 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:13:35.810382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:35.810356 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0d0152-openshift-default-98d856d9c-fz5d7"] Oct 30 06:13:36.060092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.060059 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:13:36.061084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061060 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.061084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061083 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.061290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061148 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" containerName="payload-processing" Oct 30 06:13:36.061290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061156 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" containerName="payload-processing" Oct 30 06:13:36.061290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061185 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.061290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061192 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.061501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061400 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0144da91-e849-44e2-8ac9-c8c6ad5033a5" containerName="payload-processing" Oct 30 06:13:36.061501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061420 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.061501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.061437 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" containerName="istio-proxy" Oct 30 06:13:36.066920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.066901 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.087145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.087079 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dddc53-openshift-default-dockercfg-h6dt9\"" Oct 30 06:13:36.095756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.095732 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:13:36.234138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.218982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.219037 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245297 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245345 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245404 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245484 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245664 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245887 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.254146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.245979 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kls4z\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351197 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351312 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351384 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kls4z\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351467 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351496 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351529 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351566 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351614 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.352324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.351681 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.353718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.353404 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.366072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.355605 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.366685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.366467 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.366851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.366611 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.367249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.367226 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.403434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.401689 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.403434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.402211 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.414199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.414171 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.426948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.426875 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kls4z\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\") pod \"e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:36.683137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:36.683053 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:37.021606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.020222 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:13:37.023761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.023723 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:37.023889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.023804 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:37.023889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.023852 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:13:37.471964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.471938 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:13:37.477031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.476977 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.497212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.497173 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-25d881\"" Oct 30 06:13:37.498689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.498661 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:13:37.573916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.573886 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.573916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.573922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv2k4\" (UniqueName: \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.691077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.689684 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.691077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.689734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv2k4\" (UniqueName: \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.711070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.707930 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.769623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.768989 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a9794fc-a2c6-4710-a6d9-643fcaaafb5d" path="/var/lib/kubelet/pods/8a9794fc-a2c6-4710-a6d9-643fcaaafb5d/volumes" Oct 30 06:13:37.883067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.878735 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv2k4\" (UniqueName: \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\") pod \"payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:37.902095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.892564 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerStarted","Data":"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa"} Oct 30 06:13:37.902095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.892628 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerStarted","Data":"34486dc0564ddd4ad68893fce0911d258cad1728ed7be8f139e28b8852e6e2c7"} Oct 30 06:13:37.946238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:37.946188 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podStartSLOduration=1.946174252 podStartE2EDuration="1.946174252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:37.945344407 +0000 UTC m=+2476.385721600" watchObservedRunningTime="2026-10-30 06:13:37.946174252 +0000 UTC m=+2476.386551445" Oct 30 06:13:38.109188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.108087 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:38.430931 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:38.430887 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6da28380_2c55_46e6_9d73_4656683abb32.slice/crio-0fd7341da5f1929c374a1b354130bb60f1cb6fab9598038008c84ab5f8e9abe1 WatchSource:0}: Error finding container 0fd7341da5f1929c374a1b354130bb60f1cb6fab9598038008c84ab5f8e9abe1: Status 404 returned error can't find the container with id 0fd7341da5f1929c374a1b354130bb60f1cb6fab9598038008c84ab5f8e9abe1 Oct 30 06:13:38.432414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.432385 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:13:38.504294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.504269 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:13:38.511012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.510986 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.531345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.531314 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:13:38.534438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.534187 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"openshift-service-ca.crt\"" Oct 30 06:13:38.534438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.534431 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"kube-root-ca.crt\"" Oct 30 06:13:38.534749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.534578 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"default-dockercfg-hgwk4\"" Oct 30 06:13:38.543995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.543976 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w5-91f42c\"/\"auth-test-model-91f42c-kserve-self-signed-certs\"" Oct 30 06:13:38.621097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621034 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.621314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621293 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.621400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621360 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.621490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621472 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vclx2\" (UniqueName: \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.621559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621542 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.621647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.621626 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.684085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.683994 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:38.690270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.690232 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:38.722792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.722770 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.722813 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723004 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vclx2\" (UniqueName: \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723106 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723213 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723247 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723288 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723288 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.723736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.723716 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.725915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.725889 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.726381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.726358 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.787067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.786808 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vclx2\" (UniqueName: \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\") pod \"auth-test-model-91f42c-kserve-7b57b786d5-rvdgt\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.848649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.847353 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:38.917756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.917718 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" event={"ID":"6da28380-2c55-46e6-9d73-4656683abb32","Type":"ContainerStarted","Data":"519757b1362b2ddf68dfce56d489679305596e0951b2956ba10073b4383a2117"} Oct 30 06:13:38.918025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.918006 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" event={"ID":"6da28380-2c55-46e6-9d73-4656683abb32","Type":"ContainerStarted","Data":"0fd7341da5f1929c374a1b354130bb60f1cb6fab9598038008c84ab5f8e9abe1"} Oct 30 06:13:38.918179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.918165 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:38.918328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.918317 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:38.921232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.921186 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:13:38.990293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:38.990237 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" podStartSLOduration=1.990217627 podStartE2EDuration="1.990217627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:38.980496529 +0000 UTC m=+2477.420873724" watchObservedRunningTime="2026-10-30 06:13:38.990217627 +0000 UTC m=+2477.430594823" Oct 30 06:13:39.266209 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:39.266176 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod727f5d41_57ee_4842_9a24_80385472d761.slice/crio-a60e4625d7991ddb525666266cbeffc79ef9e5171925e72a544f9aff5d535b74 WatchSource:0}: Error finding container a60e4625d7991ddb525666266cbeffc79ef9e5171925e72a544f9aff5d535b74: Status 404 returned error can't find the container with id a60e4625d7991ddb525666266cbeffc79ef9e5171925e72a544f9aff5d535b74 Oct 30 06:13:39.267887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.267857 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:13:39.676809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.676733 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5"] Oct 30 06:13:39.681951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.681928 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.701090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.698143 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5"] Oct 30 06:13:39.701422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.701401 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:13:39.701508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.701493 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:39.771979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.771630 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.771979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.771724 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.772574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.772216 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwmcm\" (UniqueName: \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.873994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.873965 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwmcm\" (UniqueName: \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.874246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.874147 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.874246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.874190 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.874801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.874779 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.876830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.876807 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.899249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.899228 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwmcm\" (UniqueName: \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\") pod \"maas-api-revoke-sub-af2bc828-67vg5\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:39.926735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.926700 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" event={"ID":"727f5d41-57ee-4842-9a24-80385472d761","Type":"ContainerStarted","Data":"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6"} Oct 30 06:13:39.926937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.926745 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" event={"ID":"727f5d41-57ee-4842-9a24-80385472d761","Type":"ContainerStarted","Data":"a60e4625d7991ddb525666266cbeffc79ef9e5171925e72a544f9aff5d535b74"} Oct 30 06:13:39.976062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:39.976004 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" podStartSLOduration=1.9759916830000002 podStartE2EDuration="1.975991683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:39.973222764 +0000 UTC m=+2478.413599957" watchObservedRunningTime="2026-10-30 06:13:39.975991683 +0000 UTC m=+2478.416368876" Oct 30 06:13:40.003939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:40.003909 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:40.313886 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:40.313852 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c66167_575f_41f5_9652_406c7f5b12d0.slice/crio-673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d WatchSource:0}: Error finding container 673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d: Status 404 returned error can't find the container with id 673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d Oct 30 06:13:40.314894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:40.314869 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5"] Oct 30 06:13:40.848242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:40.848214 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:40.948425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:40.948342 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" event={"ID":"a8c66167-575f-41f5-9652-406c7f5b12d0","Type":"ContainerStarted","Data":"1c7ca181e05f3cd69c93d805593b97a33f18d90d8d926f1d2e256c9a98b6477b"} Oct 30 06:13:40.948583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:40.948437 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" event={"ID":"a8c66167-575f-41f5-9652-406c7f5b12d0","Type":"ContainerStarted","Data":"673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d"} Oct 30 06:13:41.000344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:41.000290 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" podStartSLOduration=2.000277802 podStartE2EDuration="2.000277802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:40.99712386 +0000 UTC m=+2479.437501050" watchObservedRunningTime="2026-10-30 06:13:41.000277802 +0000 UTC m=+2479.440655028" Oct 30 06:13:41.960086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:41.960023 6171 generic.go:356] "Generic (PLEG): container finished" podID="a8c66167-575f-41f5-9652-406c7f5b12d0" containerID="1c7ca181e05f3cd69c93d805593b97a33f18d90d8d926f1d2e256c9a98b6477b" exitCode=0 Oct 30 06:13:41.960589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:41.960155 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" event={"ID":"a8c66167-575f-41f5-9652-406c7f5b12d0","Type":"ContainerDied","Data":"1c7ca181e05f3cd69c93d805593b97a33f18d90d8d926f1d2e256c9a98b6477b"} Oct 30 06:13:43.149118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.149093 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:43.287885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.287804 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\" (UniqueName: \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\") pod \"a8c66167-575f-41f5-9652-406c7f5b12d0\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " Oct 30 06:13:43.288548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.288496 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\") pod \"a8c66167-575f-41f5-9652-406c7f5b12d0\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " Oct 30 06:13:43.288933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.288900 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\") pod \"a8c66167-575f-41f5-9652-406c7f5b12d0\" (UID: \"a8c66167-575f-41f5-9652-406c7f5b12d0\") " Oct 30 06:13:43.290890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.290864 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca" pod "a8c66167-575f-41f5-9652-406c7f5b12d0" (UID: "a8c66167-575f-41f5-9652-406c7f5b12d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:43.315187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.309174 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca" pod "a8c66167-575f-41f5-9652-406c7f5b12d0" (UID: "a8c66167-575f-41f5-9652-406c7f5b12d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:43.315187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.309318 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm" pod "a8c66167-575f-41f5-9652-406c7f5b12d0" (UID: "a8c66167-575f-41f5-9652-406c7f5b12d0"). InnerVolumeSpecName "kube-api-access-dwmcm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:43.392227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.392190 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwmcm\" (UniqueName: \"kubernetes.io/projected/a8c66167-575f-41f5-9652-406c7f5b12d0-kube-api-access-dwmcm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:43.392227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.392228 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:43.392477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.392241 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c66167-575f-41f5-9652-406c7f5b12d0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:43.981133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.981095 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" event={"ID":"a8c66167-575f-41f5-9652-406c7f5b12d0","Type":"ContainerDied","Data":"673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d"} Oct 30 06:13:43.981133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.981124 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5" Oct 30 06:13:43.981363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:43.981149 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="673f0556dcf26fb5e771f7da137edec0194f8f10d1f02c634a1e13de578baf4d" Oct 30 06:13:48.848813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:48.848781 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:48.861954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:48.861930 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:49.039787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:49.039753 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:13:49.752734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:49.752695 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" podUID="42def66a-46ba-4886-abf5-244ec2ad4a38" containerName="main" containerID="cri-o://bde3a8a9fffb7358d1205e2f01739c486d6dcc3b252272e5b0d64249a2b213e5" gracePeriod=45 Oct 30 06:13:50.041924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.041893 6171 generic.go:356] "Generic (PLEG): container finished" podID="42def66a-46ba-4886-abf5-244ec2ad4a38" containerID="bde3a8a9fffb7358d1205e2f01739c486d6dcc3b252272e5b0d64249a2b213e5" exitCode=0 Oct 30 06:13:50.042396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.041971 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" event={"ID":"42def66a-46ba-4886-abf5-244ec2ad4a38","Type":"ContainerDied","Data":"bde3a8a9fffb7358d1205e2f01739c486d6dcc3b252272e5b0d64249a2b213e5"} Oct 30 06:13:50.042396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.042022 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" event={"ID":"42def66a-46ba-4886-abf5-244ec2ad4a38","Type":"ContainerDied","Data":"5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3"} Oct 30 06:13:50.042396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.042037 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b777a5fdb8e1fcde4739a78cb56f90b1c9b15adb7d5c293991022e503083ee3" Oct 30 06:13:50.054758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.054730 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:13:50.100224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100197 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100247 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100312 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\" (UniqueName: \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100395 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\" (UniqueName: \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100442 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:50.100497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100453 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100679 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:50.100991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100688 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\") pod \"42def66a-46ba-4886-abf5-244ec2ad4a38\" (UID: \"42def66a-46ba-4886-abf5-244ec2ad4a38\") " Oct 30 06:13:50.100991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.100978 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:50.101780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.101753 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.101868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.101784 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.101868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.101803 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.103107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.103083 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:50.103216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.103151 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:13:50.103216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.103197 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk" pod "42def66a-46ba-4886-abf5-244ec2ad4a38" (UID: "42def66a-46ba-4886-abf5-244ec2ad4a38"). InnerVolumeSpecName "kube-api-access-zkmnk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:50.163484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.163459 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6"] Oct 30 06:13:50.164496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164478 6171 cpu_manager.go:414] "containerMap: removing container" podUID="42def66a-46ba-4886-abf5-244ec2ad4a38" containerName="main" Oct 30 06:13:50.164496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164496 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42def66a-46ba-4886-abf5-244ec2ad4a38" containerName="main" Oct 30 06:13:50.164616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164514 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a8c66167-575f-41f5-9652-406c7f5b12d0" containerName="revoke-keys" Oct 30 06:13:50.164616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164519 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c66167-575f-41f5-9652-406c7f5b12d0" containerName="revoke-keys" Oct 30 06:13:50.164700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164653 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c66167-575f-41f5-9652-406c7f5b12d0" containerName="revoke-keys" Oct 30 06:13:50.164700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.164674 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="42def66a-46ba-4886-abf5-244ec2ad4a38" containerName="main" Oct 30 06:13:50.168670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.168654 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.187090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.186073 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:13:50.187090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.186153 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:13:50.187957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.187931 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6"] Oct 30 06:13:50.229355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.207491 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.229762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.229709 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.230196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.230157 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-729dh\" (UniqueName: \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.234580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.234535 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/42def66a-46ba-4886-abf5-244ec2ad4a38-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.234789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.234761 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/42def66a-46ba-4886-abf5-244ec2ad4a38-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.234920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.234905 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkmnk\" (UniqueName: \"kubernetes.io/projected/42def66a-46ba-4886-abf5-244ec2ad4a38-kube-api-access-zkmnk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:50.336545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.336465 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-729dh\" (UniqueName: \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.336692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.336677 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.336739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.336696 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.337480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.337457 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.339173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.339149 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.361991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.361967 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-729dh\" (UniqueName: \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\") pod \"maas-api-revoke-sub-790f8f63-nlzc6\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.488962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.488935 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:50.798348 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:13:50.798315 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode82eb8cb_48ca_4ce8_9bcf_223f7b9d2406.slice/crio-c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15 WatchSource:0}: Error finding container c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15: Status 404 returned error can't find the container with id c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15 Oct 30 06:13:50.799721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.799693 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6"] Oct 30 06:13:50.952694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:50.952664 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:13:51.055146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:51.055013 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72" Oct 30 06:13:51.055146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:51.055038 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" event={"ID":"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406","Type":"ContainerStarted","Data":"c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15"} Oct 30 06:13:51.120499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:51.120465 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:13:51.125678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:51.125655 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w6-f22a52/auth-test-model-f22a52-kserve-666964888f-tmr72"] Oct 30 06:13:51.675648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:51.675618 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42def66a-46ba-4886-abf5-244ec2ad4a38" path="/var/lib/kubelet/pods/42def66a-46ba-4886-abf5-244ec2ad4a38/volumes" Oct 30 06:13:52.068149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:52.068106 6171 generic.go:356] "Generic (PLEG): container finished" podID="e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" containerID="7edcb5ffe6a208e01812c1b796d7294ba438115ebb73be8d009ac446bef4de2f" exitCode=0 Oct 30 06:13:52.068680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:52.068151 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" event={"ID":"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406","Type":"ContainerDied","Data":"7edcb5ffe6a208e01812c1b796d7294ba438115ebb73be8d009ac446bef4de2f"} Oct 30 06:13:53.246600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.246576 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:53.387104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.387014 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\") pod \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " Oct 30 06:13:53.387309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.387294 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\" (UniqueName: \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\") pod \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " Oct 30 06:13:53.387407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.387388 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\") pod \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\" (UID: \"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406\") " Oct 30 06:13:53.387765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.387699 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca" pod "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" (UID: "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:53.387923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.387908 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:53.389234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.389209 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca" pod "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" (UID: "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:13:53.389427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.389407 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh" pod "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" (UID: "e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406"). InnerVolumeSpecName "kube-api-access-729dh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:53.489481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.489440 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-729dh\" (UniqueName: \"kubernetes.io/projected/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-kube-api-access-729dh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:53.489481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:53.489474 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:13:54.090878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:54.090847 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" Oct 30 06:13:54.091109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:54.090887 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6" event={"ID":"e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406","Type":"ContainerDied","Data":"c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15"} Oct 30 06:13:54.091109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:13:54.090942 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6067a5aad73026f0290afc17dd6d1e0d77d887f12ee2c736109409aa088ff15" Oct 30 06:13:56.382770 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:13:56.382729 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:14:00.973831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.973799 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8"] Oct 30 06:14:00.974679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.974662 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" containerName="revoke-keys" Oct 30 06:14:00.974725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.974680 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" containerName="revoke-keys" Oct 30 06:14:00.974882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.974873 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" containerName="revoke-keys" Oct 30 06:14:00.979553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.979532 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:00.996353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:00.996318 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8"] Oct 30 06:14:01.003248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.003216 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:01.003408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.003388 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:14:01.020576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.020548 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvhpj\" (UniqueName: \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.020727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.020707 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.020897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.020877 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.154067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.153698 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvhpj\" (UniqueName: \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.154067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.153856 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.154341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.154242 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.177066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.168803 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.190571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.190533 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.238939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.238858 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvhpj\" (UniqueName: \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\") pod \"maas-api-revoke-sub-96df05fe-q98v8\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.297266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.297226 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:01.701828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:01.701745 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8"] Oct 30 06:14:02.298807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:02.298767 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" event={"ID":"df5cd092-7b59-4adc-bf47-fbdfd022f67e","Type":"ContainerStarted","Data":"4ca204e414285ee69cc6842b3f9efcdcabcbec4daf523ec1ec325ff48234d4e2"} Oct 30 06:14:02.299275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:02.298832 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" event={"ID":"df5cd092-7b59-4adc-bf47-fbdfd022f67e","Type":"ContainerStarted","Data":"afbd8e1576f812e01f6b898151a23f648be504fa5bd28b0ec6bb0d9b4800ff61"} Oct 30 06:14:02.344555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:02.344452 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" podStartSLOduration=2.344438259 podStartE2EDuration="2.344438259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:02.342944586 +0000 UTC m=+2500.783321779" watchObservedRunningTime="2026-10-30 06:14:02.344438259 +0000 UTC m=+2500.784815494" Oct 30 06:14:03.309989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:03.309950 6171 generic.go:356] "Generic (PLEG): container finished" podID="df5cd092-7b59-4adc-bf47-fbdfd022f67e" containerID="4ca204e414285ee69cc6842b3f9efcdcabcbec4daf523ec1ec325ff48234d4e2" exitCode=0 Oct 30 06:14:03.310483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:03.309996 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" event={"ID":"df5cd092-7b59-4adc-bf47-fbdfd022f67e","Type":"ContainerDied","Data":"4ca204e414285ee69cc6842b3f9efcdcabcbec4daf523ec1ec325ff48234d4e2"} Oct 30 06:14:04.558226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.558199 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:04.682299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.682229 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\" (UniqueName: \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\") pod \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " Oct 30 06:14:04.682458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.682357 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\") pod \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " Oct 30 06:14:04.682515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.682458 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\") pod \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\" (UID: \"df5cd092-7b59-4adc-bf47-fbdfd022f67e\") " Oct 30 06:14:04.682863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.682838 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca" pod "df5cd092-7b59-4adc-bf47-fbdfd022f67e" (UID: "df5cd092-7b59-4adc-bf47-fbdfd022f67e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:04.683150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.683129 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:04.684576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.684557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca" pod "df5cd092-7b59-4adc-bf47-fbdfd022f67e" (UID: "df5cd092-7b59-4adc-bf47-fbdfd022f67e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:04.684676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.684580 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj" pod "df5cd092-7b59-4adc-bf47-fbdfd022f67e" (UID: "df5cd092-7b59-4adc-bf47-fbdfd022f67e"). InnerVolumeSpecName "kube-api-access-gvhpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:04.785051 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.785020 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df5cd092-7b59-4adc-bf47-fbdfd022f67e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:04.785228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:04.785067 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvhpj\" (UniqueName: \"kubernetes.io/projected/df5cd092-7b59-4adc-bf47-fbdfd022f67e-kube-api-access-gvhpj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:05.386644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:05.386597 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" event={"ID":"df5cd092-7b59-4adc-bf47-fbdfd022f67e","Type":"ContainerDied","Data":"afbd8e1576f812e01f6b898151a23f648be504fa5bd28b0ec6bb0d9b4800ff61"} Oct 30 06:14:05.386852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:05.386683 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afbd8e1576f812e01f6b898151a23f648be504fa5bd28b0ec6bb0d9b4800ff61" Oct 30 06:14:05.386852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:05.386608 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8" Oct 30 06:14:09.881431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.881393 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j"] Oct 30 06:14:09.882485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.882462 6171 cpu_manager.go:414] "containerMap: removing container" podUID="df5cd092-7b59-4adc-bf47-fbdfd022f67e" containerName="revoke-keys" Oct 30 06:14:09.882485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.882485 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df5cd092-7b59-4adc-bf47-fbdfd022f67e" containerName="revoke-keys" Oct 30 06:14:09.882692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.882678 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="df5cd092-7b59-4adc-bf47-fbdfd022f67e" containerName="revoke-keys" Oct 30 06:14:09.886980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.886961 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:09.907694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.907659 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j"] Oct 30 06:14:09.913582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.913558 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:09.913797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.913770 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:14:09.935348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.935322 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:09.935462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.935371 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:09.935735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:09.935710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhl5\" (UniqueName: \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.037218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.037183 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhl5\" (UniqueName: \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.037391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.037378 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.037455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.037416 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.038123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.038102 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.039948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.039923 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.063126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.063105 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhl5\" (UniqueName: \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\") pod \"maas-api-revoke-sub-70f0ecbd-qks5j\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.202678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.202601 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:10.515432 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:10.515405 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod912297fd_6a71_4a01_aa25_6be064421cbe.slice/crio-a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9 WatchSource:0}: Error finding container a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9: Status 404 returned error can't find the container with id a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9 Oct 30 06:14:10.516548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:10.516518 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j"] Oct 30 06:14:11.036378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.036346 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx"] Oct 30 06:14:11.042292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.042224 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-680f1658-h6tpx"] Oct 30 06:14:11.461354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.461316 6171 generic.go:356] "Generic (PLEG): container finished" podID="912297fd-6a71-4a01-aa25-6be064421cbe" containerID="934168f36c9cbb350ea1493da5617a9fdbdaebc7478037a66b2654e30b02609e" exitCode=0 Oct 30 06:14:11.461566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.461403 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" event={"ID":"912297fd-6a71-4a01-aa25-6be064421cbe","Type":"ContainerDied","Data":"934168f36c9cbb350ea1493da5617a9fdbdaebc7478037a66b2654e30b02609e"} Oct 30 06:14:11.461566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.461469 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" event={"ID":"912297fd-6a71-4a01-aa25-6be064421cbe","Type":"ContainerStarted","Data":"a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9"} Oct 30 06:14:11.528623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.528593 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv"] Oct 30 06:14:11.534027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.534008 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.561375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.561346 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv"] Oct 30 06:14:11.583335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.583302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.583486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.583385 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.583541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.583514 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzhhq\" (UniqueName: \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.676717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.676628 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad879bdb-cd1f-442d-ae9f-ca72283301ee" path="/var/lib/kubelet/pods/ad879bdb-cd1f-442d-ae9f-ca72283301ee/volumes" Oct 30 06:14:11.685322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.685296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.685450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.685349 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.685450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.685428 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzhhq\" (UniqueName: \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.686078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.686058 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.687863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.687841 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.710991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.710966 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzhhq\" (UniqueName: \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\") pod \"maas-api-revoke-sub-1805ecc1-8bkvv\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:11.867071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:11.866358 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:12.240791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.240759 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv"] Oct 30 06:14:12.491395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.491303 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" event={"ID":"f8f0485c-7edf-45fd-a01d-f449f082f785","Type":"ContainerStarted","Data":"033bb7a5da50fbfb25e7ce20f457d931db8bbf9b3cbdcb8dcfa9952408e0b7be"} Oct 30 06:14:12.671688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.671662 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:12.714306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.712487 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\") pod \"912297fd-6a71-4a01-aa25-6be064421cbe\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " Oct 30 06:14:12.714306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.712564 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\" (UniqueName: \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\") pod \"912297fd-6a71-4a01-aa25-6be064421cbe\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " Oct 30 06:14:12.714306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.712858 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\") pod \"912297fd-6a71-4a01-aa25-6be064421cbe\" (UID: \"912297fd-6a71-4a01-aa25-6be064421cbe\") " Oct 30 06:14:12.715182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.714951 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca" pod "912297fd-6a71-4a01-aa25-6be064421cbe" (UID: "912297fd-6a71-4a01-aa25-6be064421cbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:12.719397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.719326 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5" pod "912297fd-6a71-4a01-aa25-6be064421cbe" (UID: "912297fd-6a71-4a01-aa25-6be064421cbe"). InnerVolumeSpecName "kube-api-access-ckhl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:12.726534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.726092 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca" pod "912297fd-6a71-4a01-aa25-6be064421cbe" (UID: "912297fd-6a71-4a01-aa25-6be064421cbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:12.814965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.814889 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:12.814965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.814922 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/912297fd-6a71-4a01-aa25-6be064421cbe-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:12.814965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:12.814933 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhl5\" (UniqueName: \"kubernetes.io/projected/912297fd-6a71-4a01-aa25-6be064421cbe-kube-api-access-ckhl5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:13.519754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:13.514575 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" event={"ID":"912297fd-6a71-4a01-aa25-6be064421cbe","Type":"ContainerDied","Data":"a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9"} Oct 30 06:14:13.519754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:13.514626 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a948e6bd845ecd312854d3d1dc60285c9c64f5c82d4e16378f2d0949665955f9" Oct 30 06:14:13.519754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:13.514688 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j" Oct 30 06:14:13.532332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:13.532294 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" event={"ID":"f8f0485c-7edf-45fd-a01d-f449f082f785","Type":"ContainerStarted","Data":"348ddc15e5a4d5905f97bc3abaf3c5e633afc38842195bf934c40741326153ac"} Oct 30 06:14:13.582453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:13.582405 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" podStartSLOduration=2.582389819 podStartE2EDuration="2.582389819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:13.582075887 +0000 UTC m=+2512.022453079" watchObservedRunningTime="2026-10-30 06:14:13.582389819 +0000 UTC m=+2512.022767012" Oct 30 06:14:14.543600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:14.543563 6171 generic.go:356] "Generic (PLEG): container finished" podID="f8f0485c-7edf-45fd-a01d-f449f082f785" containerID="348ddc15e5a4d5905f97bc3abaf3c5e633afc38842195bf934c40741326153ac" exitCode=0 Oct 30 06:14:14.544131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:14.543605 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" event={"ID":"f8f0485c-7edf-45fd-a01d-f449f082f785","Type":"ContainerDied","Data":"348ddc15e5a4d5905f97bc3abaf3c5e633afc38842195bf934c40741326153ac"} Oct 30 06:14:15.784054 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.784026 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:15.880923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.880840 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\") pod \"f8f0485c-7edf-45fd-a01d-f449f082f785\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " Oct 30 06:14:15.881104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.880975 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\" (UniqueName: \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\") pod \"f8f0485c-7edf-45fd-a01d-f449f082f785\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " Oct 30 06:14:15.881570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.881246 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\") pod \"f8f0485c-7edf-45fd-a01d-f449f082f785\" (UID: \"f8f0485c-7edf-45fd-a01d-f449f082f785\") " Oct 30 06:14:15.881570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.881267 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca" pod "f8f0485c-7edf-45fd-a01d-f449f082f785" (UID: "f8f0485c-7edf-45fd-a01d-f449f082f785"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:15.882175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.881861 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:15.883940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.883908 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq" pod "f8f0485c-7edf-45fd-a01d-f449f082f785" (UID: "f8f0485c-7edf-45fd-a01d-f449f082f785"). InnerVolumeSpecName "kube-api-access-jzhhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:15.884281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.884256 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca" pod "f8f0485c-7edf-45fd-a01d-f449f082f785" (UID: "f8f0485c-7edf-45fd-a01d-f449f082f785"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:15.983630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.983595 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f0485c-7edf-45fd-a01d-f449f082f785-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:15.983630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:15.983626 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzhhq\" (UniqueName: \"kubernetes.io/projected/f8f0485c-7edf-45fd-a01d-f449f082f785-kube-api-access-jzhhq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:16.587641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:16.587609 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" Oct 30 06:14:16.587812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:16.587636 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv" event={"ID":"f8f0485c-7edf-45fd-a01d-f449f082f785","Type":"ContainerDied","Data":"033bb7a5da50fbfb25e7ce20f457d931db8bbf9b3cbdcb8dcfa9952408e0b7be"} Oct 30 06:14:16.587812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:16.587686 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033bb7a5da50fbfb25e7ce20f457d931db8bbf9b3cbdcb8dcfa9952408e0b7be" Oct 30 06:14:20.032459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.032417 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8"] Oct 30 06:14:20.046583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.046546 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5e8c249a-f96w8"] Oct 30 06:14:20.200941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.200911 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m"] Oct 30 06:14:20.201834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201810 6171 cpu_manager.go:414] "containerMap: removing container" podUID="912297fd-6a71-4a01-aa25-6be064421cbe" containerName="revoke-keys" Oct 30 06:14:20.201834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201831 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="912297fd-6a71-4a01-aa25-6be064421cbe" containerName="revoke-keys" Oct 30 06:14:20.202070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201858 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f8f0485c-7edf-45fd-a01d-f449f082f785" containerName="revoke-keys" Oct 30 06:14:20.202070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201864 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f0485c-7edf-45fd-a01d-f449f082f785" containerName="revoke-keys" Oct 30 06:14:20.202070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201976 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="912297fd-6a71-4a01-aa25-6be064421cbe" containerName="revoke-keys" Oct 30 06:14:20.202070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.201993 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f0485c-7edf-45fd-a01d-f449f082f785" containerName="revoke-keys" Oct 30 06:14:20.206072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.206034 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.223214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.223179 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:14:20.223652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.223632 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:14:20.227213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.227187 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m"] Oct 30 06:14:20.381156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.381072 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.381156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.381146 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.381359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.381274 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xlrp\" (UniqueName: \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.483122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.483088 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.483330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.483157 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.483330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.483232 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xlrp\" (UniqueName: \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.483963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.483939 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.485641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.485617 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.509653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.509618 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xlrp\" (UniqueName: \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\") pod \"maas-api-revoke-sub-acf0c297-wv27m\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.522109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.522086 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:20.840622 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:20.840591 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1438f908_3dc2_45ff_9bbc_977bd0ed05a6.slice/crio-bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662 WatchSource:0}: Error finding container bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662: Status 404 returned error can't find the container with id bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662 Oct 30 06:14:20.842390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:20.842360 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m"] Oct 30 06:14:21.867544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:21.863192 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e8a02e8-eeb5-4aed-910e-b52f4ed67464" path="/var/lib/kubelet/pods/1e8a02e8-eeb5-4aed-910e-b52f4ed67464/volumes" Oct 30 06:14:21.867544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:21.863740 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" event={"ID":"1438f908-3dc2-45ff-9bbc-977bd0ed05a6","Type":"ContainerStarted","Data":"9dd1fc4555fd600b013e6e5549a29f7dc5b65ca423e59877907443fb309bc4f7"} Oct 30 06:14:21.867544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:21.863806 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" event={"ID":"1438f908-3dc2-45ff-9bbc-977bd0ed05a6","Type":"ContainerStarted","Data":"bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662"} Oct 30 06:14:21.906481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:21.906429 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" podStartSLOduration=1.906415204 podStartE2EDuration="1.906415204s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:21.90492304 +0000 UTC m=+2520.345300234" watchObservedRunningTime="2026-10-30 06:14:21.906415204 +0000 UTC m=+2520.346792394" Oct 30 06:14:22.867525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:22.867489 6171 generic.go:356] "Generic (PLEG): container finished" podID="1438f908-3dc2-45ff-9bbc-977bd0ed05a6" containerID="9dd1fc4555fd600b013e6e5549a29f7dc5b65ca423e59877907443fb309bc4f7" exitCode=0 Oct 30 06:14:22.867718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:22.867577 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" event={"ID":"1438f908-3dc2-45ff-9bbc-977bd0ed05a6","Type":"ContainerDied","Data":"9dd1fc4555fd600b013e6e5549a29f7dc5b65ca423e59877907443fb309bc4f7"} Oct 30 06:14:24.050672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.050630 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:24.104895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.104862 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\" (UniqueName: \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\") pod \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " Oct 30 06:14:24.105084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.105032 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\") pod \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " Oct 30 06:14:24.105203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.105185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\") pod \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\" (UID: \"1438f908-3dc2-45ff-9bbc-977bd0ed05a6\") " Oct 30 06:14:24.105570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.105538 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca" pod "1438f908-3dc2-45ff-9bbc-977bd0ed05a6" (UID: "1438f908-3dc2-45ff-9bbc-977bd0ed05a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:24.109642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.106461 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:24.110108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.110079 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp" pod "1438f908-3dc2-45ff-9bbc-977bd0ed05a6" (UID: "1438f908-3dc2-45ff-9bbc-977bd0ed05a6"). InnerVolumeSpecName "kube-api-access-6xlrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:24.110429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.110399 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca" pod "1438f908-3dc2-45ff-9bbc-977bd0ed05a6" (UID: "1438f908-3dc2-45ff-9bbc-977bd0ed05a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:24.207811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.207731 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:24.207811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.207765 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xlrp\" (UniqueName: \"kubernetes.io/projected/1438f908-3dc2-45ff-9bbc-977bd0ed05a6-kube-api-access-6xlrp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:24.890694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.890664 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" Oct 30 06:14:24.890694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.890680 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m" event={"ID":"1438f908-3dc2-45ff-9bbc-977bd0ed05a6","Type":"ContainerDied","Data":"bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662"} Oct 30 06:14:24.890927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:24.890728 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbf2067d4bc2054c16e967dff1bf51adedf5a2e52503558fc2b2545b04d78662" Oct 30 06:14:26.382667 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:26.382631 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:14:26.988068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:26.982207 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:14:26.988068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:26.983561 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1438f908-3dc2-45ff-9bbc-977bd0ed05a6" containerName="revoke-keys" Oct 30 06:14:26.988068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:26.983585 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1438f908-3dc2-45ff-9bbc-977bd0ed05a6" containerName="revoke-keys" Oct 30 06:14:27.014325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.014271 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1438f908-3dc2-45ff-9bbc-977bd0ed05a6" containerName="revoke-keys" Oct 30 06:14:27.041083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.038517 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:14:27.041083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.038642 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.060410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.060375 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d4576\"" Oct 30 06:14:27.108012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.061223 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d4576-dockercfg-lfbl9\"" Oct 30 06:14:27.207663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.207630 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.207851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.207785 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxpsl\" (UniqueName: \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.309505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.309408 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.309658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.309529 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxpsl\" (UniqueName: \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.310184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.310161 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.335126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.335099 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxpsl\" (UniqueName: \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\") pod \"payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.454945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.454916 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:27.767799 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:27.767770 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed476ee_3ad0_4c21_b3c7_6b122fb6ceb9.slice/crio-9bb4feea0f6bfca1d9e18de4749206331ee2b4e26f3fdfdd4ba5b69b23e08c40 WatchSource:0}: Error finding container 9bb4feea0f6bfca1d9e18de4749206331ee2b4e26f3fdfdd4ba5b69b23e08c40: Status 404 returned error can't find the container with id 9bb4feea0f6bfca1d9e18de4749206331ee2b4e26f3fdfdd4ba5b69b23e08c40 Oct 30 06:14:27.769800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:27.769773 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:14:28.074428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.074340 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" event={"ID":"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9","Type":"ContainerStarted","Data":"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245"} Oct 30 06:14:28.074428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.074396 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" event={"ID":"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9","Type":"ContainerStarted","Data":"9bb4feea0f6bfca1d9e18de4749206331ee2b4e26f3fdfdd4ba5b69b23e08c40"} Oct 30 06:14:28.074428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.074425 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:28.122832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.122770 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" podStartSLOduration=2.122755019 podStartE2EDuration="2.122755019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:28.118770318 +0000 UTC m=+2526.559147512" watchObservedRunningTime="2026-10-30 06:14:28.122755019 +0000 UTC m=+2526.563132228" Oct 30 06:14:28.387321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.387247 6171 scope.go:122] "RemoveContainer" containerID="665010cdb584aa4ba2402e681ffeb0cf0589050520fc6f58905e0d5b82f91fb6" Oct 30 06:14:28.400620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.400596 6171 scope.go:122] "RemoveContainer" containerID="e12c394ad653da4e6b03510f135eb6fe2367973149e321c26d67a83bcef4bddf" Oct 30 06:14:28.412622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.412597 6171 scope.go:122] "RemoveContainer" containerID="c58ddc3db0340b3089e6d4b38358730547876b4272199168deda783fbdf5004a" Oct 30 06:14:28.423341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.423320 6171 scope.go:122] "RemoveContainer" containerID="c54d94efe203e32745c98227fa1cbdc72e9414675cae75205a4a13debd12337a" Oct 30 06:14:28.433918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.433881 6171 scope.go:122] "RemoveContainer" containerID="d2578e683b6fb8deee52ef7470fa1c30cba773557eb01dd4cb1e845295bc8b0c" Oct 30 06:14:28.444001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.443982 6171 scope.go:122] "RemoveContainer" containerID="fc5ad46bae00ce68b9be32c16a99c1b2f45ef9842f51c4e0de9a5cb74fd3a406" Oct 30 06:14:28.454946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:28.454927 6171 scope.go:122] "RemoveContainer" containerID="9d4ca657cc22109314b5f6b13013fd44926dbda88e0a67784f62a5fc49595547" Oct 30 06:14:35.034172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:35.034077 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc"] Oct 30 06:14:35.047463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:35.047430 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2d9b776-jmtdc"] Oct 30 06:14:35.676171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:35.676134 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa7d5ccb-f516-4154-af11-c258c78c20e9" path="/var/lib/kubelet/pods/aa7d5ccb-f516-4154-af11-c258c78c20e9/volumes" Oct 30 06:14:39.087655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:39.087619 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:14:40.546296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:40.546266 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 06:14:40.546873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:40.546462 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" containerID="cri-o://c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1" gracePeriod=30 Oct 30 06:14:40.563862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:40.563831 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 06:14:40.564121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:40.564083 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" containerID="cri-o://a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d" gracePeriod=30 Oct 30 06:14:41.512092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:41.512014 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:14:41.936072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:41.935976 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:14:43.073234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.073199 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:14:43.081549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.081524 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.114165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.114142 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-bf87e1-openshift-default-dockercfg-9dtb4\"" Oct 30 06:14:43.158278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.158249 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:14:43.433690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.433593 6171 patch_prober.go:29] interesting pod/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:14:43.433988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.433954 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:14:43.484653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.476453 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486147 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486221 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486323 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486461 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486552 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486659 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486783 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whfsl\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.486864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.486832 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588278 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whfsl\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588325 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588382 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588403 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588655 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588743 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588791 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588863 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.588977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.588937 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.589092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.589035 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.589291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.589272 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.589471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.589450 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.589544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.589530 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.589582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.589529 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.590826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.590809 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.590890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.590873 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.602245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.602217 6171 patch_prober.go:29] interesting pod/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:14:43.602356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.602265 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:14:43.614666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.614638 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whfsl\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.616159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.616136 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\") pod \"e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:43.697151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:43.697088 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:44.033243 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:44.033205 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1416e241_2058_4b57_bf24_3b616be2ce6a.slice/crio-3cd5578a835e43c72bf3e2496105782e7c6210ae4aa30b3c650d0dac3d647054 WatchSource:0}: Error finding container 3cd5578a835e43c72bf3e2496105782e7c6210ae4aa30b3c650d0dac3d647054: Status 404 returned error can't find the container with id 3cd5578a835e43c72bf3e2496105782e7c6210ae4aa30b3c650d0dac3d647054 Oct 30 06:14:44.036218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.036177 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:44.036352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.036272 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:44.036352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.036319 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:44.036701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.036677 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:14:44.392486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.392405 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerStarted","Data":"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686"} Oct 30 06:14:44.392486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.392456 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerStarted","Data":"3cd5578a835e43c72bf3e2496105782e7c6210ae4aa30b3c650d0dac3d647054"} Oct 30 06:14:44.451181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.451132 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podStartSLOduration=1.451118266 podStartE2EDuration="1.451118266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:44.449378777 +0000 UTC m=+2542.889755981" watchObservedRunningTime="2026-10-30 06:14:44.451118266 +0000 UTC m=+2542.891495460" Oct 30 06:14:44.697454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:44.697374 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:45.706152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.706117 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:45.906798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.901244 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 06:14:45.911350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.911320 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 06:14:45.933665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.933804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933690 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.933804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933731 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.933804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933773 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.933992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933806 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.933992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933850 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.933992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933923 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.933992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.933952 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.934270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934003 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.934270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934083 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.934270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934112 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.934270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934132 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.934270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934213 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934275 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934310 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934352 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\") pod \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\" (UID: \"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934429 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934441 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934519 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.934564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.934551 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\") pod \"75b417bf-5723-485e-bedc-c57bfe9465b9\" (UID: \"75b417bf-5723-485e-bedc-c57bfe9465b9\") " Oct 30 06:14:45.935656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.935627 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:45.935789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.935661 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:45.935984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.935959 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.936197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936176 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.936412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936391 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.936612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936589 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:45.936704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936615 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:14:45.936859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936838 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.936963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.936881 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.937078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937024 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:45.937078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937059 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.937380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937152 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.937380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937213 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "kube-api-access-h26kw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:45.937380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937261 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:45.937623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937600 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "kube-api-access-nd2zh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:45.937804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937789 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:14:45.937909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.937895 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token" pod "75b417bf-5723-485e-bedc-c57bfe9465b9" (UID: "75b417bf-5723-485e-bedc-c57bfe9465b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:45.938667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:45.938645 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy" pod "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" (UID: "acb85c8d-fe38-46bd-9eb2-8f3e37d84c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:46.049710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.049677 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/75b417bf-5723-485e-bedc-c57bfe9465b9-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050184 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050216 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050227 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050238 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nd2zh\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-kube-api-access-nd2zh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050247 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050256 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050264 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050271 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050280 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050287 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050295 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050303 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h26kw\" (UniqueName: \"kubernetes.io/projected/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-kube-api-access-h26kw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050311 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050319 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/75b417bf-5723-485e-bedc-c57bfe9465b9-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.050412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.050327 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:46.414794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.414752 6171 generic.go:356] "Generic (PLEG): container finished" podID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerID="a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d" exitCode=0 Oct 30 06:14:46.414981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.414790 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerDied","Data":"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d"} Oct 30 06:14:46.414981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.414807 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" Oct 30 06:14:46.414981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.414893 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn" event={"ID":"75b417bf-5723-485e-bedc-c57bfe9465b9","Type":"ContainerDied","Data":"f8e114d0d22a690fa772fde5068fd45802cf09c98cab12f32144e7d8a314b27a"} Oct 30 06:14:46.414981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.414928 6171 scope.go:122] "RemoveContainer" containerID="a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d" Oct 30 06:14:46.416579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.416559 6171 generic.go:356] "Generic (PLEG): container finished" podID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerID="c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1" exitCode=0 Oct 30 06:14:46.416728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.416623 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" Oct 30 06:14:46.416728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.416649 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" event={"ID":"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73","Type":"ContainerDied","Data":"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1"} Oct 30 06:14:46.416728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.416693 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl" event={"ID":"acb85c8d-fe38-46bd-9eb2-8f3e37d84c73","Type":"ContainerDied","Data":"af24ec0095207ad42bb40b4b6fc9afd9c57c6d81b8d15fddcf39cb7166ad3f56"} Oct 30 06:14:46.417153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.417123 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:46.418218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.418195 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:14:46.429281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.429264 6171 scope.go:122] "RemoveContainer" containerID="026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5" Oct 30 06:14:46.440788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.440769 6171 scope.go:122] "RemoveContainer" containerID="a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d" Oct 30 06:14:46.441081 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:46.441033 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d\": container with ID starting with a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d not found: ID does not exist" containerID="a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d" Oct 30 06:14:46.441179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.441088 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d"} err="failed to get container status \"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d\": rpc error: code = NotFound desc = could not find container \"a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d\": container with ID starting with a0e09ff71d642df0635a42c9bb0444943481a411f5e90cbdbdf8b9f1f2f4397d not found: ID does not exist" Oct 30 06:14:46.441179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.441106 6171 scope.go:122] "RemoveContainer" containerID="026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5" Oct 30 06:14:46.441344 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:46.441327 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5\": container with ID starting with 026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5 not found: ID does not exist" containerID="026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5" Oct 30 06:14:46.441393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.441349 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5"} err="failed to get container status \"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5\": rpc error: code = NotFound desc = could not find container \"026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5\": container with ID starting with 026ea6ba3714891a59f64ab1dfbbb7e446aa7d088f6f319fee0f77a3abf87da5 not found: ID does not exist" Oct 30 06:14:46.441393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.441364 6171 scope.go:122] "RemoveContainer" containerID="c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1" Oct 30 06:14:46.452299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.452278 6171 scope.go:122] "RemoveContainer" containerID="c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1" Oct 30 06:14:46.452566 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:46.452545 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1\": container with ID starting with c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1 not found: ID does not exist" containerID="c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1" Oct 30 06:14:46.452620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.452573 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1"} err="failed to get container status \"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1\": rpc error: code = NotFound desc = could not find container \"c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1\": container with ID starting with c4e30a264fa626402f82c27aead3b0bfe9e7b6eb978d08c63a3252286e4771a1 not found: ID does not exist" Oct 30 06:14:46.477621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.477599 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 06:14:46.482632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.482615 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-e2e3d0-openshift-default-57898b578c-z4qqn"] Oct 30 06:14:46.570247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.570211 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 06:14:46.575452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:46.575425 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c78066-openshift-default-5696c477f7-z48tl"] Oct 30 06:14:47.675348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:47.675316 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" path="/var/lib/kubelet/pods/75b417bf-5723-485e-bedc-c57bfe9465b9/volumes" Oct 30 06:14:47.675810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:47.675795 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" path="/var/lib/kubelet/pods/acb85c8d-fe38-46bd-9eb2-8f3e37d84c73/volumes" Oct 30 06:14:48.056315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:48.056282 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f"] Oct 30 06:14:48.069925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:48.069897 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ee8e44e-5d72f"] Oct 30 06:14:49.238818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.238780 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:14:49.252036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.252001 6171 cpu_manager.go:414] "containerMap: removing container" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:49.252250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.252232 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:49.252364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.252350 6171 cpu_manager.go:414] "containerMap: removing container" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" Oct 30 06:14:49.252462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.252451 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" Oct 30 06:14:49.252830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.252814 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:49.262595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.262574 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:49.408070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.392113 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="acb85c8d-fe38-46bd-9eb2-8f3e37d84c73" containerName="istio-proxy" Oct 30 06:14:49.500902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.500645 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:14:49.500902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.500767 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.528757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.525777 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-dfc3d2-openshift-default-dockercfg-96hvf\"" Oct 30 06:14:49.598609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.598579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.598812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.598639 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.598878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.598800 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.598878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.598853 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.598996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.598977 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.599148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.599135 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.599204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.599192 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.599257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.599213 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.599257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.599241 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92xrt\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.674745 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.674716 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c1ead80-a52a-4bd4-80fc-d815712b7b40" path="/var/lib/kubelet/pods/2c1ead80-a52a-4bd4-80fc-d815712b7b40/volumes" Oct 30 06:14:49.700547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700512 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700554 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700604 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700628 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700650 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700767 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700834 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700865 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700907 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92xrt\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.700960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700924 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.701283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.700978 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.701283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.701084 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.701283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.701118 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.701458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.701401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.703641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.703619 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.703641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.703633 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.726456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.726430 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92xrt\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.726977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.726956 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\") pod \"e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:49.878778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:49.878688 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:50.197132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.192390 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:14:50.197132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.195358 6171 cpu_manager.go:414] "containerMap: removing container" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:50.197132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.195395 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75b417bf-5723-485e-bedc-c57bfe9465b9" containerName="istio-proxy" Oct 30 06:14:50.209100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.202003 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.224250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.219429 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf87e1-dockercfg-64bjr\"" Oct 30 06:14:50.224596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.220270 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf87e1\"" Oct 30 06:14:50.255282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.255255 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:14:50.264884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.264855 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.265032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.264902 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqwj8\" (UniqueName: \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.311430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.292811 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:14:50.395070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.391410 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.395070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.391481 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqwj8\" (UniqueName: \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.395070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.393161 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.471790 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:50.465750 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdd9bf3_ea1c_4b74_a551_b12e83748a7d.slice/crio-8d6f0e387b0147e51d30efa84e285b1ec938d017742e1fa622752544e3b491c1 WatchSource:0}: Error finding container 8d6f0e387b0147e51d30efa84e285b1ec938d017742e1fa622752544e3b491c1: Status 404 returned error can't find the container with id 8d6f0e387b0147e51d30efa84e285b1ec938d017742e1fa622752544e3b491c1 Oct 30 06:14:50.471790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.469107 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqwj8\" (UniqueName: \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\") pod \"payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.490362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.490309 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:50.490552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.490418 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:50.490552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.490493 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:14:50.528391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.528110 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerStarted","Data":"8d6f0e387b0147e51d30efa84e285b1ec938d017742e1fa622752544e3b491c1"} Oct 30 06:14:50.555083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.555033 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:50.893328 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:50.893295 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcff9f80c_25cf_494a_a0db_4d5290197e30.slice/crio-5ca43ef0a110c2b179533684efe4bed569bd4fb54b68bd901ddea876cfb4c65d WatchSource:0}: Error finding container 5ca43ef0a110c2b179533684efe4bed569bd4fb54b68bd901ddea876cfb4c65d: Status 404 returned error can't find the container with id 5ca43ef0a110c2b179533684efe4bed569bd4fb54b68bd901ddea876cfb4c65d Oct 30 06:14:50.894425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:50.894394 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:14:51.539416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.539381 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" event={"ID":"cff9f80c-25cf-494a-a0db-4d5290197e30","Type":"ContainerStarted","Data":"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a"} Oct 30 06:14:51.540112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.539439 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" event={"ID":"cff9f80c-25cf-494a-a0db-4d5290197e30","Type":"ContainerStarted","Data":"5ca43ef0a110c2b179533684efe4bed569bd4fb54b68bd901ddea876cfb4c65d"} Oct 30 06:14:51.540112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.539470 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:14:51.541350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.541324 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerStarted","Data":"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f"} Oct 30 06:14:51.589695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.589651 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" podStartSLOduration=1.589638721 podStartE2EDuration="1.589638721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:51.584502363 +0000 UTC m=+2550.024879557" watchObservedRunningTime="2026-10-30 06:14:51.589638721 +0000 UTC m=+2550.030015914" Oct 30 06:14:51.635527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.635474 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podStartSLOduration=2.635461259 podStartE2EDuration="2.635461259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:51.633259537 +0000 UTC m=+2550.073636731" watchObservedRunningTime="2026-10-30 06:14:51.635461259 +0000 UTC m=+2550.075838452" Oct 30 06:14:51.699675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.699643 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:14:51.893373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:51.884514 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:52.005715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:52.001693 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:52.551296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:52.551256 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:52.552240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:52.552223 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:14:55.879092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.879023 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:14:55.885786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.885762 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:55.886207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.886187 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:14:55.900130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.895625 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:55.915848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.915817 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:14:55.925083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.922628 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dfc3d2\"" Oct 30 06:14:55.925083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:55.922825 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:14:56.068069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.067559 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl8xm\" (UniqueName: \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.068069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.067706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-828fm\" (UniqueName: \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.068069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.067840 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.068069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.067903 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.172032 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl8xm\" (UniqueName: \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.172206 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-828fm\" (UniqueName: \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.172347 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.172408 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.173566 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.178147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.174848 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.230072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.212889 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-828fm\" (UniqueName: \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.231295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.231198 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl8xm\" (UniqueName: \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\") pod \"payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.383764 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:56.383724 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:14:56.515661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.515366 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:56.521489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.521448 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:56.535405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.535356 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" podUID="c1d70593-fbfb-42ec-acb0-429cd5952483" containerName="main" containerID="cri-o://ca60b2edd7a28dff2d66bbabb75db63d22d3ab11a98b3b1b70849c2de274e93d" gracePeriod=45 Oct 30 06:14:56.706434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.706369 6171 generic.go:356] "Generic (PLEG): container finished" podID="c1d70593-fbfb-42ec-acb0-429cd5952483" containerID="ca60b2edd7a28dff2d66bbabb75db63d22d3ab11a98b3b1b70849c2de274e93d" exitCode=0 Oct 30 06:14:56.706580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.706438 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" event={"ID":"c1d70593-fbfb-42ec-acb0-429cd5952483","Type":"ContainerDied","Data":"ca60b2edd7a28dff2d66bbabb75db63d22d3ab11a98b3b1b70849c2de274e93d"} Oct 30 06:14:56.834770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.834748 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:14:56.855238 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:56.855212 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8164b629_629e_4c82_885e_ce6b6a0c29f9.slice/crio-2a5f5430b056ed23b3f3cfd3a8e183e49009bb8885eac871aa22efae0c7ea6b9 WatchSource:0}: Error finding container 2a5f5430b056ed23b3f3cfd3a8e183e49009bb8885eac871aa22efae0c7ea6b9: Status 404 returned error can't find the container with id 2a5f5430b056ed23b3f3cfd3a8e183e49009bb8885eac871aa22efae0c7ea6b9 Oct 30 06:14:56.858883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.858860 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:14:56.876221 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:14:56.876195 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d0948a_d43f_4bee_9f6b_d06d990a7d2f.slice/crio-24e763b111b486bc5185e101bd0d89505f111dcbea174e86f0e672967be3b527 WatchSource:0}: Error finding container 24e763b111b486bc5185e101bd0d89505f111dcbea174e86f0e672967be3b527: Status 404 returned error can't find the container with id 24e763b111b486bc5185e101bd0d89505f111dcbea174e86f0e672967be3b527 Oct 30 06:14:56.878985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.878962 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:14:56.958205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.958168 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" containerName="main" containerID="cri-o://c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33" gracePeriod=45 Oct 30 06:14:56.960910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:56.960882 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" containerName="main" probeResult="failure" output="Get \"https://10.133.1.55:8000/ready\": dial tcp 10.133.1.55:8000: connect: connection refused" Oct 30 06:14:57.002645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.002612 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.002753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.002700 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.002847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.002831 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.002900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.002868 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.002962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.002900 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.003113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.003095 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\" (UniqueName: \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.003193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.003156 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\") pod \"c1d70593-fbfb-42ec-acb0-429cd5952483\" (UID: \"c1d70593-fbfb-42ec-acb0-429cd5952483\") " Oct 30 06:14:57.003254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.003208 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.003538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.003500 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.004085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.004061 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.004175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.004090 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.004175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.004105 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.005211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.005183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.005307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.005250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:57.005307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.005272 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7" pod "c1d70593-fbfb-42ec-acb0-429cd5952483" (UID: "c1d70593-fbfb-42ec-acb0-429cd5952483"). InnerVolumeSpecName "kube-api-access-xr4l7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:57.105623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.105588 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr4l7\" (UniqueName: \"kubernetes.io/projected/c1d70593-fbfb-42ec-acb0-429cd5952483-kube-api-access-xr4l7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.105623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.105617 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1d70593-fbfb-42ec-acb0-429cd5952483-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.105623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.105628 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1d70593-fbfb-42ec-acb0-429cd5952483-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.257127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.257098 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.307630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\" (UniqueName: \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.307748 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.307787 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308155 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308178 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308208 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\" (UniqueName: \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308355 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308442 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\") pod \"024caeea-1005-42e4-88a3-3da505b79aa7\" (UID: \"024caeea-1005-42e4-88a3-3da505b79aa7\") " Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.308797 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.309992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.309963 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.310724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.310007 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.310724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.310067 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.327543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.327513 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "kube-api-access-r7lph". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:57.328083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.328030 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:14:57.328327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.328306 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm" pod "024caeea-1005-42e4-88a3-3da505b79aa7" (UID: "024caeea-1005-42e4-88a3-3da505b79aa7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:57.411244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.411213 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7lph\" (UniqueName: \"kubernetes.io/projected/024caeea-1005-42e4-88a3-3da505b79aa7-kube-api-access-r7lph\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.411346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.411255 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/024caeea-1005-42e4-88a3-3da505b79aa7-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.411346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.411266 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/024caeea-1005-42e4-88a3-3da505b79aa7-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:14:57.717619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.717588 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" event={"ID":"8164b629-629e-4c82-885e-ce6b6a0c29f9","Type":"ContainerStarted","Data":"5fa3bdb9b4be7b2447a0abfbdc986d0e3839c35842fe215792c8c276de161510"} Oct 30 06:14:57.717779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.717649 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" event={"ID":"8164b629-629e-4c82-885e-ce6b6a0c29f9","Type":"ContainerStarted","Data":"2a5f5430b056ed23b3f3cfd3a8e183e49009bb8885eac871aa22efae0c7ea6b9"} Oct 30 06:14:57.717779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.717694 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:14:57.719326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.719298 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" Oct 30 06:14:57.719464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.719344 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj" event={"ID":"c1d70593-fbfb-42ec-acb0-429cd5952483","Type":"ContainerDied","Data":"883096d99e81a1a0c985d42e6f9888f03d14ec3eef2ac69202e38a06a9def847"} Oct 30 06:14:57.719464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.719388 6171 scope.go:122] "RemoveContainer" containerID="ca60b2edd7a28dff2d66bbabb75db63d22d3ab11a98b3b1b70849c2de274e93d" Oct 30 06:14:57.721417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.721391 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" event={"ID":"99d0948a-d43f-4bee-9f6b-d06d990a7d2f","Type":"ContainerStarted","Data":"65a82a65c79916992a4b412acf34c1dbe1d979b8751ebfc88e6c47d36fc2b9b7"} Oct 30 06:14:57.721538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.721516 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:14:57.721655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.721546 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" event={"ID":"99d0948a-d43f-4bee-9f6b-d06d990a7d2f","Type":"ContainerStarted","Data":"24e763b111b486bc5185e101bd0d89505f111dcbea174e86f0e672967be3b527"} Oct 30 06:14:57.723119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.723094 6171 generic.go:356] "Generic (PLEG): container finished" podID="024caeea-1005-42e4-88a3-3da505b79aa7" containerID="c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33" exitCode=0 Oct 30 06:14:57.723245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.723131 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" Oct 30 06:14:57.723245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.723224 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" event={"ID":"024caeea-1005-42e4-88a3-3da505b79aa7","Type":"ContainerDied","Data":"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33"} Oct 30 06:14:57.723352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.723279 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf" event={"ID":"024caeea-1005-42e4-88a3-3da505b79aa7","Type":"ContainerDied","Data":"9117e5c50d6d24d93ec7f8c0ad70f27390ba1c2d98ec376f18b455378596593a"} Oct 30 06:14:57.733192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.733166 6171 scope.go:122] "RemoveContainer" containerID="c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33" Oct 30 06:14:57.745647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.745625 6171 scope.go:122] "RemoveContainer" containerID="c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33" Oct 30 06:14:57.745953 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:14:57.745930 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33\": container with ID starting with c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33 not found: ID does not exist" containerID="c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33" Oct 30 06:14:57.746028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.745962 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33"} err="failed to get container status \"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33\": rpc error: code = NotFound desc = could not find container \"c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33\": container with ID starting with c31f4858343248c7f84082766be7d6adb671b5fa5e2c7acc5f9a075611423e33 not found: ID does not exist" Oct 30 06:14:57.765929 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.765885 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" podStartSLOduration=2.765873817 podStartE2EDuration="2.765873817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:57.762252066 +0000 UTC m=+2556.202629261" watchObservedRunningTime="2026-10-30 06:14:57.765873817 +0000 UTC m=+2556.206251029" Oct 30 06:14:57.813959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.813938 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:14:57.822208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.822187 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-cc33d3/sub-test-model-cc33d3-kserve-864cbdd4cf-wn7mj"] Oct 30 06:14:57.861029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.861001 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:14:57.868368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.868345 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6b5b1e/auth-test-model-6b5b1e-kserve-86bb474b5-j64xf"] Oct 30 06:14:57.952122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:57.947782 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" podStartSLOduration=2.947761822 podStartE2EDuration="2.947761822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:57.916659687 +0000 UTC m=+2556.357036881" watchObservedRunningTime="2026-10-30 06:14:57.947761822 +0000 UTC m=+2556.388139016" Oct 30 06:14:59.674453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:59.674415 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" path="/var/lib/kubelet/pods/024caeea-1005-42e4-88a3-3da505b79aa7/volumes" Oct 30 06:14:59.674997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:14:59.674981 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d70593-fbfb-42ec-acb0-429cd5952483" path="/var/lib/kubelet/pods/c1d70593-fbfb-42ec-acb0-429cd5952483/volumes" Oct 30 06:15:00.161602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.161560 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:00.163008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.162982 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c1d70593-fbfb-42ec-acb0-429cd5952483" containerName="main" Oct 30 06:15:00.163008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.163007 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d70593-fbfb-42ec-acb0-429cd5952483" containerName="main" Oct 30 06:15:00.163246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.163100 6171 cpu_manager.go:414] "containerMap: removing container" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" containerName="main" Oct 30 06:15:00.163246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.163108 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" containerName="main" Oct 30 06:15:00.163359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.163303 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="024caeea-1005-42e4-88a3-3da505b79aa7" containerName="main" Oct 30 06:15:00.163359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.163327 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d70593-fbfb-42ec-acb0-429cd5952483" containerName="main" Oct 30 06:15:00.251620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.237242 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:00.251620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.237911 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.251620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.247296 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.253981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.253264 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:00.263525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.262776 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.265112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.264595 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:00.272492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.271573 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:00.273626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.273258 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:15:00.273990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.273854 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.282591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.282334 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:00.282591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.282435 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.287664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.287644 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:00.299626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.299596 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:00.306474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.306403 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:00.321083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.317386 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:00.331749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331722 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvxks\" (UniqueName: \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.331898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331825 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.331898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331852 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg9gr\" (UniqueName: \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.331898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331883 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.332117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331927 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.332117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.331967 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gp7p\" (UniqueName: \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.332117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.332006 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.332298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.332153 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.332298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.332188 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m29px\" (UniqueName: \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.332298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.332216 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78rs4\" (UniqueName: \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.356277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.356245 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:00.358580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.358554 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.489797 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.489897 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gp7p\" (UniqueName: \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.489977 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490161 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490214 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m29px\" (UniqueName: \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490315 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78rs4\" (UniqueName: \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490392 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvxks\" (UniqueName: \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490648 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490686 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg9gr\" (UniqueName: \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.491249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.490932 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.507195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.507163 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.514710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.514312 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.514861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.514833 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.515836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.515813 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.517536 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.517507 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:00.517651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.517561 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:00.517651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.517590 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:00.517762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.517739 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.518803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.518780 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.525444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.525061 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:00.525444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.525235 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.530630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.530608 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:00.530738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.530706 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.541914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.541884 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg9gr\" (UniqueName: \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.545982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.542949 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gp7p\" (UniqueName: \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.545982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.542988 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:00.545982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.544180 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvxks\" (UniqueName: \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.545982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.544812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78rs4\" (UniqueName: \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.545982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.545931 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m29px\" (UniqueName: \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.547351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.547332 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:00.552888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.552873 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.556209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.556189 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:00.561581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.561564 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.563968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.563951 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:00.578855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.578836 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:00.581029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.581003 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:00.590125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.590102 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:00.604811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.604747 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:00.606632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.606494 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.606807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.606780 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtfvh\" (UniqueName: \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.647371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.647338 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:00.658090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.657395 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710106 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtfvh\" (UniqueName: \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710186 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710268 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mpvd\" (UniqueName: \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710447 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j75f\" (UniqueName: \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710511 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mvp\" (UniqueName: \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710575 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710614 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpj6h\" (UniqueName: \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710765 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710835 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710899 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.710971 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p685f\" (UniqueName: \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.724012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.713440 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.774814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.774780 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtfvh\" (UniqueName: \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\") pod \"maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.814863 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j75f\" (UniqueName: \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.814928 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mvp\" (UniqueName: \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.814986 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpj6h\" (UniqueName: \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815127 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815210 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815277 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815350 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p685f\" (UniqueName: \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815432 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.815482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815470 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.816208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.815495 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mpvd\" (UniqueName: \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.817229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.816439 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.817229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.816733 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.817229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.817141 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.817549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.817492 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.817549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.817505 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.819286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.819264 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:00.826539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.826504 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:00.846524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.846108 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mvp\" (UniqueName: \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\") pod \"maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:00.848443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.848400 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p685f\" (UniqueName: \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\") pod \"maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:00.851339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.851317 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j75f\" (UniqueName: \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\") pod \"maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:00.851683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.851663 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpj6h\" (UniqueName: \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:00.851739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.851662 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mpvd\" (UniqueName: \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\") pod \"maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:00.948387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.946697 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:00.980426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:00.980183 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:01.012518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.012490 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:01.028811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.028784 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:01.066066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.062036 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:01.141227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.141191 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:01.214204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.212821 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:01.252112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.250724 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:01.259253 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:01.259200 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae2fbf77_9ad7_492d_8908_7dedfa97665c.slice/crio-00cf2a24a4303f188789e7508f0bcb800f090b5dbb84cf4347b4caf46ad9e2f5 WatchSource:0}: Error finding container 00cf2a24a4303f188789e7508f0bcb800f090b5dbb84cf4347b4caf46ad9e2f5: Status 404 returned error can't find the container with id 00cf2a24a4303f188789e7508f0bcb800f090b5dbb84cf4347b4caf46ad9e2f5 Oct 30 06:15:01.264399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.262327 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:01.276419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.276391 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:01.379308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.378628 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:01.387022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.386397 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:01.476928 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:01.476896 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7ab0c7_25ea_42c1_9a86_07b5abc968ce.slice/crio-6a40b750da78c529f615c98e68ce62eed466ce51f178bcf1db660c2889ee87e6 WatchSource:0}: Error finding container 6a40b750da78c529f615c98e68ce62eed466ce51f178bcf1db660c2889ee87e6: Status 404 returned error can't find the container with id 6a40b750da78c529f615c98e68ce62eed466ce51f178bcf1db660c2889ee87e6 Oct 30 06:15:01.487153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.482026 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:01.576070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.576027 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:01.764846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.764733 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:01.773153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:01.773128 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:02.003067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.001128 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerStarted","Data":"dc3c94c6c7b5e7430c7213f9bf592be075ed5c9a657ed86947b8d0175cdc839c"} Oct 30 06:15:02.007069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.004906 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerStarted","Data":"61f58372a5f2306ecdd47495a6af6cefe33783a04facb7130106c064ee234515"} Oct 30 06:15:02.007247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.007067 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerStarted","Data":"6a40b750da78c529f615c98e68ce62eed466ce51f178bcf1db660c2889ee87e6"} Oct 30 06:15:02.014069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.010606 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerStarted","Data":"abaccffad97088239ce28bba685eb8eb41d05302d40e54e0fa6f3ba121e3dbd3"} Oct 30 06:15:02.014069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.010657 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerStarted","Data":"25f1dbba6527d2aec294ca306b15ead3dcf208c1de74f57f34ed5a38a813012d"} Oct 30 06:15:02.014069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.013717 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerStarted","Data":"00cf2a24a4303f188789e7508f0bcb800f090b5dbb84cf4347b4caf46ad9e2f5"} Oct 30 06:15:02.015734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.015704 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerStarted","Data":"46cb5d2877f805a41089890a9c878ae217b650ff0853dcf8a0eef4504d4ecca2"} Oct 30 06:15:02.033639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.033603 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerStarted","Data":"981972761e7010661b55e272320cffefb0dd7a9c0189d07527c129f63d5e5fd4"} Oct 30 06:15:02.050319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.050287 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerStarted","Data":"80af537b5fca284d8125b2f5eba0b7d095c3c66f95a496f95d13a27e4842eade"} Oct 30 06:15:02.052251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.052221 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerStarted","Data":"59f7fc24c8175a76093810f8795416a53da5898cb969cdec5e45132637454cdf"} Oct 30 06:15:02.052490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.052466 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerStarted","Data":"1dd1ad5bf0c3c7b73fb07004996fd185b6d92e434d9017d327bcc33c1b501b70"} Oct 30 06:15:02.053985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.053961 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerStarted","Data":"6d188d6cef0dd288fc805b065ac48775af9db827cf3b38431981401facd0bb1c"} Oct 30 06:15:02.055697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.055676 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerStarted","Data":"23de0fdfd51e35e050a32e62846aee7f47b03ad7a7ef60a946758336cbec8a19"} Oct 30 06:15:02.077350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.077287 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" podStartSLOduration=2.077267467 podStartE2EDuration="2.077267467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.071537949 +0000 UTC m=+2560.511915142" watchObservedRunningTime="2026-10-30 06:15:02.077267467 +0000 UTC m=+2560.517644665" Oct 30 06:15:02.123891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.123832 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" podStartSLOduration=2.123814207 podStartE2EDuration="2.123814207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:02.123721742 +0000 UTC m=+2560.564098935" watchObservedRunningTime="2026-10-30 06:15:02.123814207 +0000 UTC m=+2560.564191403" Oct 30 06:15:02.567251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:02.567217 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:15:03.069716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.069686 6171 generic.go:356] "Generic (PLEG): container finished" podID="7f801879-e408-4b05-acac-db0dac8e0f41" containerID="abaccffad97088239ce28bba685eb8eb41d05302d40e54e0fa6f3ba121e3dbd3" exitCode=60 Oct 30 06:15:03.070176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.069763 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerDied","Data":"abaccffad97088239ce28bba685eb8eb41d05302d40e54e0fa6f3ba121e3dbd3"} Oct 30 06:15:03.070176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.070144 6171 scope.go:122] "RemoveContainer" containerID="abaccffad97088239ce28bba685eb8eb41d05302d40e54e0fa6f3ba121e3dbd3" Oct 30 06:15:03.071802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.071775 6171 generic.go:356] "Generic (PLEG): container finished" podID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerID="9fa8508ac9d95bc251b1c7abea2abb31fe17f9e5e61799f60823e17c914cecad" exitCode=60 Oct 30 06:15:03.071917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.071847 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerDied","Data":"9fa8508ac9d95bc251b1c7abea2abb31fe17f9e5e61799f60823e17c914cecad"} Oct 30 06:15:03.072164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.072147 6171 scope.go:122] "RemoveContainer" containerID="9fa8508ac9d95bc251b1c7abea2abb31fe17f9e5e61799f60823e17c914cecad" Oct 30 06:15:03.073628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.073588 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerStarted","Data":"f92dcbd580698c55b1f9bb9650351c691e3a11235bd6e3eb71a6e67f6954fcea"} Oct 30 06:15:03.075243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.075223 6171 generic.go:356] "Generic (PLEG): container finished" podID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerID="1761876e0624322d295652e197a6f675d3d6a73af47b5b2e5a68af31609d1356" exitCode=60 Oct 30 06:15:03.075351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.075258 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerDied","Data":"1761876e0624322d295652e197a6f675d3d6a73af47b5b2e5a68af31609d1356"} Oct 30 06:15:03.075535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.075518 6171 scope.go:122] "RemoveContainer" containerID="1761876e0624322d295652e197a6f675d3d6a73af47b5b2e5a68af31609d1356" Oct 30 06:15:03.077014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.076991 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerStarted","Data":"4b3b33dea598e16afb7e1ec62fc5d7aee2ed9325982d16d716e1a776b701783d"} Oct 30 06:15:03.078594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.078573 6171 generic.go:356] "Generic (PLEG): container finished" podID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerID="59f7fc24c8175a76093810f8795416a53da5898cb969cdec5e45132637454cdf" exitCode=60 Oct 30 06:15:03.078690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.078650 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerDied","Data":"59f7fc24c8175a76093810f8795416a53da5898cb969cdec5e45132637454cdf"} Oct 30 06:15:03.078848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.078833 6171 scope.go:122] "RemoveContainer" containerID="59f7fc24c8175a76093810f8795416a53da5898cb969cdec5e45132637454cdf" Oct 30 06:15:03.107079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.100797 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerStarted","Data":"3281c2d318ab4b6292bb6b11cdea3a40886d35505610a36eb2f3f475700b7dfa"} Oct 30 06:15:03.115923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.115328 6171 generic.go:356] "Generic (PLEG): container finished" podID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerID="1fab00f8a7cb56905b07d34eaf92c82fdb1467d666f05f0de9ec3a9d417ab1a8" exitCode=60 Oct 30 06:15:03.117408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.117383 6171 scope.go:122] "RemoveContainer" containerID="1fab00f8a7cb56905b07d34eaf92c82fdb1467d666f05f0de9ec3a9d417ab1a8" Oct 30 06:15:03.119495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.118553 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerDied","Data":"1fab00f8a7cb56905b07d34eaf92c82fdb1467d666f05f0de9ec3a9d417ab1a8"} Oct 30 06:15:03.149832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.149802 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerStarted","Data":"f3b15b6a8bf3014156cb7940a9d6de8cf511f783580b097c905234be50471268"} Oct 30 06:15:03.167584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.150520 6171 scope.go:122] "RemoveContainer" containerID="f3b15b6a8bf3014156cb7940a9d6de8cf511f783580b097c905234be50471268" Oct 30 06:15:03.179375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.179323 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" podStartSLOduration=3.179307429 podStartE2EDuration="3.179307429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.176232244 +0000 UTC m=+2561.616609438" watchObservedRunningTime="2026-10-30 06:15:03.179307429 +0000 UTC m=+2561.619684622" Oct 30 06:15:03.353291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.353189 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" podStartSLOduration=3.353176881 podStartE2EDuration="3.353176881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.346639215 +0000 UTC m=+2561.787016411" watchObservedRunningTime="2026-10-30 06:15:03.353176881 +0000 UTC m=+2561.793554091" Oct 30 06:15:03.413008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:03.412957 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" podStartSLOduration=3.41293786 podStartE2EDuration="3.41293786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.409219418 +0000 UTC m=+2561.849596625" watchObservedRunningTime="2026-10-30 06:15:03.41293786 +0000 UTC m=+2561.853315053" Oct 30 06:15:04.169344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.168948 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerStarted","Data":"e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9"} Oct 30 06:15:04.173352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.172457 6171 generic.go:356] "Generic (PLEG): container finished" podID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerID="3281c2d318ab4b6292bb6b11cdea3a40886d35505610a36eb2f3f475700b7dfa" exitCode=60 Oct 30 06:15:04.173352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.172527 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerDied","Data":"3281c2d318ab4b6292bb6b11cdea3a40886d35505610a36eb2f3f475700b7dfa"} Oct 30 06:15:04.173352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.172784 6171 scope.go:122] "RemoveContainer" containerID="3281c2d318ab4b6292bb6b11cdea3a40886d35505610a36eb2f3f475700b7dfa" Oct 30 06:15:04.177105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.177079 6171 generic.go:356] "Generic (PLEG): container finished" podID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerID="7dcdc234d3a1eb0760ef82518153f8c0a92c873f641513745012551525d9c851" exitCode=60 Oct 30 06:15:04.177234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.177158 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerDied","Data":"7dcdc234d3a1eb0760ef82518153f8c0a92c873f641513745012551525d9c851"} Oct 30 06:15:04.177458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.177399 6171 scope.go:122] "RemoveContainer" containerID="7dcdc234d3a1eb0760ef82518153f8c0a92c873f641513745012551525d9c851" Oct 30 06:15:04.223423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.223187 6171 generic.go:356] "Generic (PLEG): container finished" podID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerID="42450af8a35645c58e3015def6d9a4d92bb0016fb6adae71921f6d2eb6ae2ccd" exitCode=60 Oct 30 06:15:04.223423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.223356 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerDied","Data":"42450af8a35645c58e3015def6d9a4d92bb0016fb6adae71921f6d2eb6ae2ccd"} Oct 30 06:15:04.224283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.223800 6171 scope.go:122] "RemoveContainer" containerID="42450af8a35645c58e3015def6d9a4d92bb0016fb6adae71921f6d2eb6ae2ccd" Oct 30 06:15:04.227886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.227498 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerStarted","Data":"8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79"} Oct 30 06:15:04.231570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.231547 6171 generic.go:356] "Generic (PLEG): container finished" podID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerID="f3b15b6a8bf3014156cb7940a9d6de8cf511f783580b097c905234be50471268" exitCode=60 Oct 30 06:15:04.231956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.231939 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerDied","Data":"f3b15b6a8bf3014156cb7940a9d6de8cf511f783580b097c905234be50471268"} Oct 30 06:15:04.232159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.232135 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerStarted","Data":"e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec"} Oct 30 06:15:04.241595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.240844 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerStarted","Data":"b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b"} Oct 30 06:15:04.248821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.248278 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerStarted","Data":"3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04"} Oct 30 06:15:04.253635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.252710 6171 generic.go:356] "Generic (PLEG): container finished" podID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerID="f92dcbd580698c55b1f9bb9650351c691e3a11235bd6e3eb71a6e67f6954fcea" exitCode=60 Oct 30 06:15:04.253635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.252807 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerDied","Data":"f92dcbd580698c55b1f9bb9650351c691e3a11235bd6e3eb71a6e67f6954fcea"} Oct 30 06:15:04.253635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.253207 6171 scope.go:122] "RemoveContainer" containerID="f92dcbd580698c55b1f9bb9650351c691e3a11235bd6e3eb71a6e67f6954fcea" Oct 30 06:15:04.257916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.257888 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerStarted","Data":"e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3"} Oct 30 06:15:04.268929 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.268897 6171 generic.go:356] "Generic (PLEG): container finished" podID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerID="4b3b33dea598e16afb7e1ec62fc5d7aee2ed9325982d16d716e1a776b701783d" exitCode=60 Oct 30 06:15:04.269236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.269196 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerDied","Data":"4b3b33dea598e16afb7e1ec62fc5d7aee2ed9325982d16d716e1a776b701783d"} Oct 30 06:15:04.270984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.270965 6171 scope.go:122] "RemoveContainer" containerID="4b3b33dea598e16afb7e1ec62fc5d7aee2ed9325982d16d716e1a776b701783d" Oct 30 06:15:04.371674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.371629 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" podStartSLOduration=4.371616337 podStartE2EDuration="4.371616337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.364825954 +0000 UTC m=+2562.805203146" watchObservedRunningTime="2026-10-30 06:15:04.371616337 +0000 UTC m=+2562.811993552" Oct 30 06:15:04.961807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.961766 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:15:04.971550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.971524 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:04.972194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.972132 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:15:04.976586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.976542 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" podStartSLOduration=4.976528436 podStartE2EDuration="4.976528436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.79398266 +0000 UTC m=+2563.234359853" watchObservedRunningTime="2026-10-30 06:15:04.976528436 +0000 UTC m=+2563.416905628" Oct 30 06:15:04.992019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:04.987841 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" podStartSLOduration=4.987822631 podStartE2EDuration="4.987822631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.860479276 +0000 UTC m=+2563.300856471" watchObservedRunningTime="2026-10-30 06:15:04.987822631 +0000 UTC m=+2563.428199826" Oct 30 06:15:05.014155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.012219 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-f6302c\"" Oct 30 06:15:05.014155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.012355 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-f6302c-dockercfg-wkq6s\"" Oct 30 06:15:05.024713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.024666 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" podStartSLOduration=5.024648679 podStartE2EDuration="5.024648679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.991774186 +0000 UTC m=+2563.432151381" watchObservedRunningTime="2026-10-30 06:15:05.024648679 +0000 UTC m=+2563.465025873" Oct 30 06:15:05.070689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.070654 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.070944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.070922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbnsh\" (UniqueName: \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.173087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.173058 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.173547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.173531 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbnsh\" (UniqueName: \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.173691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.173669 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.244485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.244423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbnsh\" (UniqueName: \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\") pod \"payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.311098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.306785 6171 generic.go:356] "Generic (PLEG): container finished" podID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerID="3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04" exitCode=60 Oct 30 06:15:05.311098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.306912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerDied","Data":"3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04"} Oct 30 06:15:05.311098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.306968 6171 scope.go:122] "RemoveContainer" containerID="9fa8508ac9d95bc251b1c7abea2abb31fe17f9e5e61799f60823e17c914cecad" Oct 30 06:15:05.311098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.307555 6171 scope.go:122] "RemoveContainer" containerID="3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04" Oct 30 06:15:05.312633 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.308032 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l_odh-ai-gateway-infra(ae2fbf77-9ad7-492d-8908-7dedfa97665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" Oct 30 06:15:05.319339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.319312 6171 generic.go:356] "Generic (PLEG): container finished" podID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerID="e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3" exitCode=60 Oct 30 06:15:05.319454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.319341 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerDied","Data":"e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3"} Oct 30 06:15:05.319758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.319678 6171 scope.go:122] "RemoveContainer" containerID="e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3" Oct 30 06:15:05.320153 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.320012 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww_odh-ai-gateway-infra(5d0fd932-4293-4670-a1e7-ffea84b24d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" Oct 30 06:15:05.322714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.322689 6171 generic.go:356] "Generic (PLEG): container finished" podID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerID="e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9" exitCode=60 Oct 30 06:15:05.322808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.322770 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerDied","Data":"e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9"} Oct 30 06:15:05.323601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.323578 6171 scope.go:122] "RemoveContainer" containerID="e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9" Oct 30 06:15:05.323928 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.323901 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px_odh-ai-gateway-infra(bbb83f9d-19b3-4a42-947a-081e19c55622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" Oct 30 06:15:05.327472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.327453 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerStarted","Data":"f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1"} Oct 30 06:15:05.333355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.333339 6171 scope.go:122] "RemoveContainer" containerID="1761876e0624322d295652e197a6f675d3d6a73af47b5b2e5a68af31609d1356" Oct 30 06:15:05.333835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.333816 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerStarted","Data":"687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18"} Oct 30 06:15:05.335672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.335529 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:05.336002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.335984 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerStarted","Data":"b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674"} Oct 30 06:15:05.338459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.338440 6171 generic.go:356] "Generic (PLEG): container finished" podID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerID="8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79" exitCode=60 Oct 30 06:15:05.338554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.338541 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerDied","Data":"8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79"} Oct 30 06:15:05.338693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.338679 6171 scope.go:122] "RemoveContainer" containerID="8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79" Oct 30 06:15:05.338945 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.338924 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm_odh-ai-gateway-infra(a49655a2-0273-48ad-a6ae-6751667e17ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" Oct 30 06:15:05.340591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.340568 6171 generic.go:356] "Generic (PLEG): container finished" podID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerID="e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec" exitCode=60 Oct 30 06:15:05.340674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.340645 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerDied","Data":"e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec"} Oct 30 06:15:05.341128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.341105 6171 scope.go:122] "RemoveContainer" containerID="e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec" Oct 30 06:15:05.341370 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.341351 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm_odh-ai-gateway-infra(ee7ab0c7-25ea-42c1-9a86-07b5abc968ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" Oct 30 06:15:05.342681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.342655 6171 generic.go:356] "Generic (PLEG): container finished" podID="7f801879-e408-4b05-acac-db0dac8e0f41" containerID="b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b" exitCode=60 Oct 30 06:15:05.342749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.342717 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerDied","Data":"b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b"} Oct 30 06:15:05.343025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.343004 6171 scope.go:122] "RemoveContainer" containerID="b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b" Oct 30 06:15:05.343411 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:05.343386 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn_odh-ai-gateway-infra(7f801879-e408-4b05-acac-db0dac8e0f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" Oct 30 06:15:05.357023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.356995 6171 scope.go:122] "RemoveContainer" containerID="59f7fc24c8175a76093810f8795416a53da5898cb969cdec5e45132637454cdf" Oct 30 06:15:05.416876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.416846 6171 scope.go:122] "RemoveContainer" containerID="1fab00f8a7cb56905b07d34eaf92c82fdb1467d666f05f0de9ec3a9d417ab1a8" Oct 30 06:15:05.440968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.440744 6171 scope.go:122] "RemoveContainer" containerID="f3b15b6a8bf3014156cb7940a9d6de8cf511f783580b097c905234be50471268" Oct 30 06:15:05.467568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.467536 6171 scope.go:122] "RemoveContainer" containerID="abaccffad97088239ce28bba685eb8eb41d05302d40e54e0fa6f3ba121e3dbd3" Oct 30 06:15:05.714835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.714775 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" podStartSLOduration=5.714755965 podStartE2EDuration="5.714755965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:05.70919016 +0000 UTC m=+2564.149567354" watchObservedRunningTime="2026-10-30 06:15:05.714755965 +0000 UTC m=+2564.155133159" Oct 30 06:15:05.720596 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:05.720563 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3de532ed_e289_4524_85ad_9e42592c7147.slice/crio-64a00a25672ec28d88a1a6f5bf8c5c30dfed9726112d34406d85adb89ce91222 WatchSource:0}: Error finding container 64a00a25672ec28d88a1a6f5bf8c5c30dfed9726112d34406d85adb89ce91222: Status 404 returned error can't find the container with id 64a00a25672ec28d88a1a6f5bf8c5c30dfed9726112d34406d85adb89ce91222 Oct 30 06:15:05.725906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.725883 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:15:05.941552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:05.941492 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" podStartSLOduration=5.94147929 podStartE2EDuration="5.94147929s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:05.935589256 +0000 UTC m=+2564.375966451" watchObservedRunningTime="2026-10-30 06:15:05.94147929 +0000 UTC m=+2564.381856482" Oct 30 06:15:06.460199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.460121 6171 scope.go:122] "RemoveContainer" containerID="b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b" Oct 30 06:15:06.460971 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.460945 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn_odh-ai-gateway-infra(7f801879-e408-4b05-acac-db0dac8e0f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" Oct 30 06:15:06.463868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.463831 6171 scope.go:122] "RemoveContainer" containerID="3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04" Oct 30 06:15:06.464529 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.464505 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l_odh-ai-gateway-infra(ae2fbf77-9ad7-492d-8908-7dedfa97665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" Oct 30 06:15:06.473141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.473109 6171 generic.go:356] "Generic (PLEG): container finished" podID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerID="e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087" exitCode=60 Oct 30 06:15:06.473588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.473573 6171 scope.go:122] "RemoveContainer" containerID="e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087" Oct 30 06:15:06.474063 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.473913 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg_odh-ai-gateway-infra(1375bbc9-4ccb-4173-bde3-8b9e22f8d731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" Oct 30 06:15:06.474063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.473929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerDied","Data":"e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087"} Oct 30 06:15:06.474063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.474014 6171 scope.go:122] "RemoveContainer" containerID="f92dcbd580698c55b1f9bb9650351c691e3a11235bd6e3eb71a6e67f6954fcea" Oct 30 06:15:06.476687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.476218 6171 scope.go:122] "RemoveContainer" containerID="e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3" Oct 30 06:15:06.476869 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.476842 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww_odh-ai-gateway-infra(5d0fd932-4293-4670-a1e7-ffea84b24d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" Oct 30 06:15:06.478183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.478161 6171 generic.go:356] "Generic (PLEG): container finished" podID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerID="d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c" exitCode=60 Oct 30 06:15:06.478275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.478226 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerDied","Data":"d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c"} Oct 30 06:15:06.478468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.478453 6171 scope.go:122] "RemoveContainer" containerID="d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c" Oct 30 06:15:06.479437 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.478684 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz_odh-ai-gateway-infra(a11e045d-1571-44f4-843a-67cad6d0d07f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" Oct 30 06:15:06.505069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.480246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" event={"ID":"3de532ed-e289-4524-85ad-9e42592c7147","Type":"ContainerStarted","Data":"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a"} Oct 30 06:15:06.505069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.480293 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" event={"ID":"3de532ed-e289-4524-85ad-9e42592c7147","Type":"ContainerStarted","Data":"64a00a25672ec28d88a1a6f5bf8c5c30dfed9726112d34406d85adb89ce91222"} Oct 30 06:15:06.505069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.480901 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:06.510367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.510342 6171 scope.go:122] "RemoveContainer" containerID="e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9" Oct 30 06:15:06.510741 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.510717 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px_odh-ai-gateway-infra(bbb83f9d-19b3-4a42-947a-081e19c55622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" Oct 30 06:15:06.513637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.513611 6171 generic.go:356] "Generic (PLEG): container finished" podID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerID="f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1" exitCode=60 Oct 30 06:15:06.513824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.513797 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerDied","Data":"f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1"} Oct 30 06:15:06.514318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.514297 6171 scope.go:122] "RemoveContainer" containerID="f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1" Oct 30 06:15:06.514932 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.514904 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp_odh-ai-gateway-infra(9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" Oct 30 06:15:06.516756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.516732 6171 generic.go:356] "Generic (PLEG): container finished" podID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerID="687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18" exitCode=60 Oct 30 06:15:06.516864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.516808 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerDied","Data":"687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18"} Oct 30 06:15:06.517886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.517256 6171 scope.go:122] "RemoveContainer" containerID="687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18" Oct 30 06:15:06.517886 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.517734 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65_odh-ai-gateway-infra(81e40a21-5725-4637-bbc6-31bb193af0cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" Oct 30 06:15:06.519205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.519187 6171 generic.go:356] "Generic (PLEG): container finished" podID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerID="b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674" exitCode=60 Oct 30 06:15:06.519308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.519287 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerDied","Data":"b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674"} Oct 30 06:15:06.519536 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.519517 6171 scope.go:122] "RemoveContainer" containerID="b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674" Oct 30 06:15:06.519884 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.519863 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt_odh-ai-gateway-infra(f38bd9ab-ce56-41ea-978d-faa7f57f8a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" Oct 30 06:15:06.521494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.521479 6171 scope.go:122] "RemoveContainer" containerID="8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79" Oct 30 06:15:06.521635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.521616 6171 scope.go:122] "RemoveContainer" containerID="4b3b33dea598e16afb7e1ec62fc5d7aee2ed9325982d16d716e1a776b701783d" Oct 30 06:15:06.521832 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.521809 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm_odh-ai-gateway-infra(a49655a2-0273-48ad-a6ae-6751667e17ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" Oct 30 06:15:06.524263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.524247 6171 scope.go:122] "RemoveContainer" containerID="e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec" Oct 30 06:15:06.524534 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:06.524513 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm_odh-ai-gateway-infra(ee7ab0c7-25ea-42c1-9a86-07b5abc968ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" Oct 30 06:15:06.540105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.540077 6171 scope.go:122] "RemoveContainer" containerID="3281c2d318ab4b6292bb6b11cdea3a40886d35505610a36eb2f3f475700b7dfa" Oct 30 06:15:06.566646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.566620 6171 scope.go:122] "RemoveContainer" containerID="7dcdc234d3a1eb0760ef82518153f8c0a92c873f641513745012551525d9c851" Oct 30 06:15:06.607965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.606559 6171 scope.go:122] "RemoveContainer" containerID="42450af8a35645c58e3015def6d9a4d92bb0016fb6adae71921f6d2eb6ae2ccd" Oct 30 06:15:06.757012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:06.756959 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" podStartSLOduration=2.7569458879999997 podStartE2EDuration="2.756945888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:06.75232956 +0000 UTC m=+2565.192706754" watchObservedRunningTime="2026-10-30 06:15:06.756945888 +0000 UTC m=+2565.197323080" Oct 30 06:15:07.543079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:07.543009 6171 scope.go:122] "RemoveContainer" containerID="e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087" Oct 30 06:15:07.551965 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:07.551929 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg_odh-ai-gateway-infra(1375bbc9-4ccb-4173-bde3-8b9e22f8d731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" Oct 30 06:15:07.556555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:07.556526 6171 scope.go:122] "RemoveContainer" containerID="d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c" Oct 30 06:15:07.557165 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:07.557139 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz_odh-ai-gateway-infra(a11e045d-1571-44f4-843a-67cad6d0d07f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" Oct 30 06:15:07.558742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:07.558722 6171 scope.go:122] "RemoveContainer" containerID="f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1" Oct 30 06:15:07.559002 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:07.558978 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp_odh-ai-gateway-infra(9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" Oct 30 06:15:07.560909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:07.560889 6171 scope.go:122] "RemoveContainer" containerID="687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18" Oct 30 06:15:07.561217 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:07.561192 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65_odh-ai-gateway-infra(81e40a21-5725-4637-bbc6-31bb193af0cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" Oct 30 06:15:07.563713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:07.563691 6171 scope.go:122] "RemoveContainer" containerID="b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674" Oct 30 06:15:07.563908 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:07.563888 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt_odh-ai-gateway-infra(f38bd9ab-ce56-41ea-978d-faa7f57f8a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" Oct 30 06:15:08.863351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:08.863261 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:15:08.864000 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:08.863983 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:15:09.970344 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:09.960369 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:15:09.970938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:09.970876 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:15:13.634840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.634806 6171 generic.go:356] "Generic (PLEG): container finished" podID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerID="8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa" exitCode=0 Oct 30 06:15:13.635322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.634878 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerDied","Data":"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa"} Oct 30 06:15:13.635373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.635322 6171 scope.go:122] "RemoveContainer" containerID="8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa" Oct 30 06:15:13.637143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.637110 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:13.637252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.637183 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:13.637252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:13.637215 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:14.648013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:14.647975 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerStarted","Data":"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433"} Oct 30 06:15:14.699474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:14.699434 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:15:14.699840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:14.699817 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:14.700818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:14.700789 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:15.664883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:15.664854 6171 scope.go:122] "RemoveContainer" containerID="e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9" Oct 30 06:15:15.665423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:15.665008 6171 scope.go:122] "RemoveContainer" containerID="8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79" Oct 30 06:15:15.665423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:15.665166 6171 scope.go:122] "RemoveContainer" containerID="b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b" Oct 30 06:15:15.706580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:15.706546 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:15.706732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:15.706602 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:16.664931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.664900 6171 scope.go:122] "RemoveContainer" containerID="e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec" Oct 30 06:15:16.671515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.671487 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerStarted","Data":"ce29a4323a711f220291e607ed575a9ba6f3ce840c135212bc15077c0bf2e21a"} Oct 30 06:15:16.673532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.673491 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerStarted","Data":"eac2a62aeb4697997a20e21e0e438a952fc1471227ee0f368822b73a0e7a610c"} Oct 30 06:15:16.675701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.675676 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerStarted","Data":"260237f6736b2365a92151b57038d8952ed7d2a36a663c693f6cfefc9efba534"} Oct 30 06:15:16.705087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.694336 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:16.705087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:16.694383 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:17.664456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.664364 6171 scope.go:122] "RemoveContainer" containerID="687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18" Oct 30 06:15:17.664646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.664523 6171 scope.go:122] "RemoveContainer" containerID="3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04" Oct 30 06:15:17.664721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.664684 6171 scope.go:122] "RemoveContainer" containerID="e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3" Oct 30 06:15:17.665123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.665087 6171 scope.go:122] "RemoveContainer" containerID="b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674" Oct 30 06:15:17.698076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.692008 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:17.710090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.699128 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:17.718449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.718415 6171 generic.go:356] "Generic (PLEG): container finished" podID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerID="eac2a62aeb4697997a20e21e0e438a952fc1471227ee0f368822b73a0e7a610c" exitCode=60 Oct 30 06:15:17.718602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.718471 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerDied","Data":"eac2a62aeb4697997a20e21e0e438a952fc1471227ee0f368822b73a0e7a610c"} Oct 30 06:15:17.718602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.718535 6171 scope.go:122] "RemoveContainer" containerID="e82ddc145e9e0ba0880859a1b32af0e80e4a5f5770350015b84e19923c5d03d9" Oct 30 06:15:17.719391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.719210 6171 scope.go:122] "RemoveContainer" containerID="eac2a62aeb4697997a20e21e0e438a952fc1471227ee0f368822b73a0e7a610c" Oct 30 06:15:17.719696 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:17.719669 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px_odh-ai-gateway-infra(bbb83f9d-19b3-4a42-947a-081e19c55622)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" Oct 30 06:15:17.721642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.721622 6171 generic.go:356] "Generic (PLEG): container finished" podID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerID="260237f6736b2365a92151b57038d8952ed7d2a36a663c693f6cfefc9efba534" exitCode=60 Oct 30 06:15:17.721760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.721695 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerDied","Data":"260237f6736b2365a92151b57038d8952ed7d2a36a663c693f6cfefc9efba534"} Oct 30 06:15:17.722583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.722563 6171 scope.go:122] "RemoveContainer" containerID="260237f6736b2365a92151b57038d8952ed7d2a36a663c693f6cfefc9efba534" Oct 30 06:15:17.723005 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:17.722974 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm_odh-ai-gateway-infra(a49655a2-0273-48ad-a6ae-6751667e17ff)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" Oct 30 06:15:17.724301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.724272 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerStarted","Data":"9e5be38f604a5cdfa6891f4ec3c1bd3fcdc95873697901339a1829055f167567"} Oct 30 06:15:17.727926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.727905 6171 generic.go:356] "Generic (PLEG): container finished" podID="7f801879-e408-4b05-acac-db0dac8e0f41" containerID="ce29a4323a711f220291e607ed575a9ba6f3ce840c135212bc15077c0bf2e21a" exitCode=60 Oct 30 06:15:17.728085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.728070 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerDied","Data":"ce29a4323a711f220291e607ed575a9ba6f3ce840c135212bc15077c0bf2e21a"} Oct 30 06:15:17.729790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.728373 6171 scope.go:122] "RemoveContainer" containerID="ce29a4323a711f220291e607ed575a9ba6f3ce840c135212bc15077c0bf2e21a" Oct 30 06:15:17.729790 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:17.728623 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn_odh-ai-gateway-infra(7f801879-e408-4b05-acac-db0dac8e0f41)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" Oct 30 06:15:17.750118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.750096 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:17.818057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.816886 6171 scope.go:122] "RemoveContainer" containerID="8a3efd89868400589efa76454ae37893584c90306aac056510defde6a9e7eb79" Oct 30 06:15:17.833317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.833289 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:17.845989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.845965 6171 scope.go:122] "RemoveContainer" containerID="b6e42da1c80ddc1b4567ebd175e6bdc69493c81bdec9bbcdb11603751536691b" Oct 30 06:15:17.923662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:17.923578 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:18.579185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.579114 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:15:18.702068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.693237 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:18.702068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.693289 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:18.739667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.739640 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerStarted","Data":"ba65936d653f2260c09c9034d72b23f4b2383ee0e6eafe49648aeab4792ac103"} Oct 30 06:15:18.750736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.749089 6171 generic.go:356] "Generic (PLEG): container finished" podID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerID="9e5be38f604a5cdfa6891f4ec3c1bd3fcdc95873697901339a1829055f167567" exitCode=60 Oct 30 06:15:18.750736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.749304 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerDied","Data":"9e5be38f604a5cdfa6891f4ec3c1bd3fcdc95873697901339a1829055f167567"} Oct 30 06:15:18.750736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.749356 6171 scope.go:122] "RemoveContainer" containerID="e96c0d327a2d6002836f1bbe354f9cc302c8f424a26527f2bc33cc71dd588eec" Oct 30 06:15:18.750736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.749822 6171 scope.go:122] "RemoveContainer" containerID="9e5be38f604a5cdfa6891f4ec3c1bd3fcdc95873697901339a1829055f167567" Oct 30 06:15:18.750736 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:18.750441 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm_odh-ai-gateway-infra(ee7ab0c7-25ea-42c1-9a86-07b5abc968ce)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" Oct 30 06:15:18.763145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.763116 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerStarted","Data":"df0837ef65ab72004fa0056a9ae361839ebdc6e65f034df3494248d909429467"} Oct 30 06:15:18.766022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.765986 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerStarted","Data":"d2e84cfaf7239e66e287866205e510a2fa4137a31d8f9d581241377b3e57e835"} Oct 30 06:15:18.771450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.771421 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerStarted","Data":"c77be04f9afe420ee5503c4ff15effb12a14252f5a38764e41db216bd24315b5"} Oct 30 06:15:18.857605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:18.856248 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:19.348650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.348624 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:19.353413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.353394 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:19.359512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.359464 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:19.462289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.462253 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\") pod \"a49655a2-0273-48ad-a6ae-6751667e17ff\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " Oct 30 06:15:19.462656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.462631 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca" pod "a49655a2-0273-48ad-a6ae-6751667e17ff" (UID: "a49655a2-0273-48ad-a6ae-6751667e17ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:19.462826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.462807 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\" (UniqueName: \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\") pod \"bbb83f9d-19b3-4a42-947a-081e19c55622\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " Oct 30 06:15:19.462934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.462919 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\") pod \"7f801879-e408-4b05-acac-db0dac8e0f41\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " Oct 30 06:15:19.463079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.463031 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\" (UniqueName: \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\") pod \"7f801879-e408-4b05-acac-db0dac8e0f41\" (UID: \"7f801879-e408-4b05-acac-db0dac8e0f41\") " Oct 30 06:15:19.463133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.463105 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\" (UniqueName: \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\") pod \"a49655a2-0273-48ad-a6ae-6751667e17ff\" (UID: \"a49655a2-0273-48ad-a6ae-6751667e17ff\") " Oct 30 06:15:19.463189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.463175 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\") pod \"bbb83f9d-19b3-4a42-947a-081e19c55622\" (UID: \"bbb83f9d-19b3-4a42-947a-081e19c55622\") " Oct 30 06:15:19.464265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.464235 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca" pod "bbb83f9d-19b3-4a42-947a-081e19c55622" (UID: "bbb83f9d-19b3-4a42-947a-081e19c55622"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:19.464657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.464634 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca" pod "7f801879-e408-4b05-acac-db0dac8e0f41" (UID: "7f801879-e408-4b05-acac-db0dac8e0f41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:19.465466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.465439 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f801879-e408-4b05-acac-db0dac8e0f41-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.465466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.465467 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bbb83f9d-19b3-4a42-947a-081e19c55622-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.465633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.465480 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a49655a2-0273-48ad-a6ae-6751667e17ff-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.466788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.466759 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px" pod "a49655a2-0273-48ad-a6ae-6751667e17ff" (UID: "a49655a2-0273-48ad-a6ae-6751667e17ff"). InnerVolumeSpecName "kube-api-access-m29px". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:19.467229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.467205 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr" pod "7f801879-e408-4b05-acac-db0dac8e0f41" (UID: "7f801879-e408-4b05-acac-db0dac8e0f41"). InnerVolumeSpecName "kube-api-access-dg9gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:19.470159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.470136 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks" pod "bbb83f9d-19b3-4a42-947a-081e19c55622" (UID: "bbb83f9d-19b3-4a42-947a-081e19c55622"). InnerVolumeSpecName "kube-api-access-xvxks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:19.567399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.567366 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg9gr\" (UniqueName: \"kubernetes.io/projected/7f801879-e408-4b05-acac-db0dac8e0f41-kube-api-access-dg9gr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.567399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.567399 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m29px\" (UniqueName: \"kubernetes.io/projected/a49655a2-0273-48ad-a6ae-6751667e17ff-kube-api-access-m29px\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.567662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.567415 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvxks\" (UniqueName: \"kubernetes.io/projected/bbb83f9d-19b3-4a42-947a-081e19c55622-kube-api-access-xvxks\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:19.665962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.665874 6171 scope.go:122] "RemoveContainer" containerID="d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c" Oct 30 06:15:19.708363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.708321 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" start-of-body= Oct 30 06:15:19.708813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.708385 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.74:15021/healthz/ready\": dial tcp 10.133.1.74:15021: connect: connection refused" Oct 30 06:15:19.788497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.788463 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" event={"ID":"7f801879-e408-4b05-acac-db0dac8e0f41","Type":"ContainerDied","Data":"25f1dbba6527d2aec294ca306b15ead3dcf208c1de74f57f34ed5a38a813012d"} Oct 30 06:15:19.788734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.788709 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn" Oct 30 06:15:19.788841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.788823 6171 scope.go:122] "RemoveContainer" containerID="ce29a4323a711f220291e607ed575a9ba6f3ce840c135212bc15077c0bf2e21a" Oct 30 06:15:19.793451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.793411 6171 generic.go:356] "Generic (PLEG): container finished" podID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerID="df0837ef65ab72004fa0056a9ae361839ebdc6e65f034df3494248d909429467" exitCode=60 Oct 30 06:15:19.793586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.793487 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerDied","Data":"df0837ef65ab72004fa0056a9ae361839ebdc6e65f034df3494248d909429467"} Oct 30 06:15:19.793955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.793936 6171 scope.go:122] "RemoveContainer" containerID="df0837ef65ab72004fa0056a9ae361839ebdc6e65f034df3494248d909429467" Oct 30 06:15:19.794351 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:19.794326 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l_odh-ai-gateway-infra(ae2fbf77-9ad7-492d-8908-7dedfa97665c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" Oct 30 06:15:19.796148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.796124 6171 generic.go:356] "Generic (PLEG): container finished" podID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerID="d2e84cfaf7239e66e287866205e510a2fa4137a31d8f9d581241377b3e57e835" exitCode=60 Oct 30 06:15:19.796247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.796190 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerDied","Data":"d2e84cfaf7239e66e287866205e510a2fa4137a31d8f9d581241377b3e57e835"} Oct 30 06:15:19.796717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.796607 6171 scope.go:122] "RemoveContainer" containerID="d2e84cfaf7239e66e287866205e510a2fa4137a31d8f9d581241377b3e57e835" Oct 30 06:15:19.796948 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:19.796929 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww_odh-ai-gateway-infra(5d0fd932-4293-4670-a1e7-ffea84b24d29)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" Oct 30 06:15:19.798506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.798477 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" Oct 30 06:15:19.798506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.798493 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px" event={"ID":"bbb83f9d-19b3-4a42-947a-081e19c55622","Type":"ContainerDied","Data":"1dd1ad5bf0c3c7b73fb07004996fd185b6d92e434d9017d327bcc33c1b501b70"} Oct 30 06:15:19.802127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.802108 6171 generic.go:356] "Generic (PLEG): container finished" podID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerID="c77be04f9afe420ee5503c4ff15effb12a14252f5a38764e41db216bd24315b5" exitCode=60 Oct 30 06:15:19.802252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.802186 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerDied","Data":"c77be04f9afe420ee5503c4ff15effb12a14252f5a38764e41db216bd24315b5"} Oct 30 06:15:19.802690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.802671 6171 scope.go:122] "RemoveContainer" containerID="c77be04f9afe420ee5503c4ff15effb12a14252f5a38764e41db216bd24315b5" Oct 30 06:15:19.803076 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:19.803019 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65_odh-ai-gateway-infra(81e40a21-5725-4637-bbc6-31bb193af0cf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" Oct 30 06:15:19.804355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.804322 6171 generic.go:356] "Generic (PLEG): container finished" podID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerID="ba65936d653f2260c09c9034d72b23f4b2383ee0e6eafe49648aeab4792ac103" exitCode=60 Oct 30 06:15:19.804468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.804393 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerDied","Data":"ba65936d653f2260c09c9034d72b23f4b2383ee0e6eafe49648aeab4792ac103"} Oct 30 06:15:19.804760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.804745 6171 scope.go:122] "RemoveContainer" containerID="ba65936d653f2260c09c9034d72b23f4b2383ee0e6eafe49648aeab4792ac103" Oct 30 06:15:19.804990 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:19.804973 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt_odh-ai-gateway-infra(f38bd9ab-ce56-41ea-978d-faa7f57f8a09)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" Oct 30 06:15:19.805286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.805270 6171 scope.go:122] "RemoveContainer" containerID="3d0836f3ec9dcf17619f9bd6427b4e287483c432dbe0e701176db00e0700da04" Oct 30 06:15:19.807667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.806682 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" Oct 30 06:15:19.807667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.806727 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm" event={"ID":"a49655a2-0273-48ad-a6ae-6751667e17ff","Type":"ContainerDied","Data":"61f58372a5f2306ecdd47495a6af6cefe33783a04facb7130106c064ee234515"} Oct 30 06:15:19.809301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.809279 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:19.824968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.824946 6171 scope.go:122] "RemoveContainer" containerID="e41e582aa73d2ea7f788905b10cdaf84f2164dc02eb439283ceb93e2f0a91fb3" Oct 30 06:15:19.843535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.843505 6171 scope.go:122] "RemoveContainer" containerID="eac2a62aeb4697997a20e21e0e438a952fc1471227ee0f368822b73a0e7a610c" Oct 30 06:15:19.864113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.863459 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:19.881531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.880282 6171 scope.go:122] "RemoveContainer" containerID="687298ec5149bbb5fec8b304432b5a25f72c99245a87935266935a388b9a0f18" Oct 30 06:15:19.898092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.897945 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-91f42c-29889015-hf2px"] Oct 30 06:15:19.923900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.923534 6171 scope.go:122] "RemoveContainer" containerID="b8b94054c3d1ed36ac434bb7e5268a993bbb5ad6a98433ccb270b8adb3a53674" Oct 30 06:15:19.926835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.926567 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:19.961711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.961653 6171 scope.go:122] "RemoveContainer" containerID="260237f6736b2365a92151b57038d8952ed7d2a36a663c693f6cfefc9efba534" Oct 30 06:15:19.973516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.973475 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:19.988129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:19.983667 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-cfe14f-29889015-brndn"] Oct 30 06:15:20.054105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.053782 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:20.101420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.091449 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:20.101420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.096911 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:20.192682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.192657 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\") pod \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " Oct 30 06:15:20.192939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.192919 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\" (UniqueName: \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\") pod \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\" (UID: \"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce\") " Oct 30 06:15:20.193746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.193720 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca" pod "ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" (UID: "ee7ab0c7-25ea-42c1-9a86-07b5abc968ce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:20.193916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.193885 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.197934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.197896 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f" pod "ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" (UID: "ee7ab0c7-25ea-42c1-9a86-07b5abc968ce"). InnerVolumeSpecName "kube-api-access-p685f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:20.243597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.243566 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:20.252721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.252696 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-d1bfce-29889015-vm5cm"] Oct 30 06:15:20.295763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.295732 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p685f\" (UniqueName: \"kubernetes.io/projected/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce-kube-api-access-p685f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:20.664231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.664199 6171 scope.go:122] "RemoveContainer" containerID="f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1" Oct 30 06:15:20.664645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.664624 6171 scope.go:122] "RemoveContainer" containerID="e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087" Oct 30 06:15:20.677569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.677542 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:15:20.679009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.678984 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679009 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679028 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679059 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679100 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679109 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679125 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679132 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679142 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679148 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679160 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679167 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679182 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679189 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679230 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679237 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679268 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679274 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679545 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679563 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679575 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679586 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679594 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679604 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679613 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679630 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679641 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:20.679719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.679658 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:20.693572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.693263 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:15:20.694816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.694795 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.695504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.695487 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.699723 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.701620 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.701655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.701787 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdq5\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.701964 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.702101 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.703010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.703099 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.703200 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.704084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.703289 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.709107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.709084 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:15:20.804794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.804759 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.804952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.804843 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.804952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.804870 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.804995 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805113 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805247 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805260 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805277 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805289 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805412 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdq5\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805463 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805653 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.805961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.805912 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.806122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.806028 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.830475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.830450 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" Oct 30 06:15:20.830578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.830500 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm" event={"ID":"ee7ab0c7-25ea-42c1-9a86-07b5abc968ce","Type":"ContainerDied","Data":"6a40b750da78c529f615c98e68ce62eed466ce51f178bcf1db660c2889ee87e6"} Oct 30 06:15:20.830578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.830570 6171 scope.go:122] "RemoveContainer" containerID="9e5be38f604a5cdfa6891f4ec3c1bd3fcdc95873697901339a1829055f167567" Oct 30 06:15:20.839361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.839335 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerStarted","Data":"a33b8adf873e9fa63dc41529faec3470dcd64d91d1c376047cdf88dc6dfd163f"} Oct 30 06:15:20.884526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.884493 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.884746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.884725 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.900770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.900734 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:20.900906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:20.900857 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdq5\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\") pod \"e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:21.066578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.066534 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:21.134742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.134711 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:21.196072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.196018 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:21.198404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.198227 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:21.210790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.210763 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-1354a5-29889015-4nkmm"] Oct 30 06:15:21.213166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.212588 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\") pod \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " Oct 30 06:15:21.213166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.212776 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\" (UniqueName: \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\") pod \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\" (UID: \"f38bd9ab-ce56-41ea-978d-faa7f57f8a09\") " Oct 30 06:15:21.213166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.212899 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\" (UniqueName: \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\") pod \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " Oct 30 06:15:21.213166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.212946 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\") pod \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\" (UID: \"ae2fbf77-9ad7-492d-8908-7dedfa97665c\") " Oct 30 06:15:21.214258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.214223 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca" pod "f38bd9ab-ce56-41ea-978d-faa7f57f8a09" (UID: "f38bd9ab-ce56-41ea-978d-faa7f57f8a09"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.214258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.214230 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca" pod "ae2fbf77-9ad7-492d-8908-7dedfa97665c" (UID: "ae2fbf77-9ad7-492d-8908-7dedfa97665c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.218652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.217791 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p" pod "ae2fbf77-9ad7-492d-8908-7dedfa97665c" (UID: "ae2fbf77-9ad7-492d-8908-7dedfa97665c"). InnerVolumeSpecName "kube-api-access-9gp7p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.224749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.224719 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f" pod "f38bd9ab-ce56-41ea-978d-faa7f57f8a09" (UID: "f38bd9ab-ce56-41ea-978d-faa7f57f8a09"). InnerVolumeSpecName "kube-api-access-2j75f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.275972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.275928 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:21.386137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.386096 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\") pod \"81e40a21-5725-4637-bbc6-31bb193af0cf\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " Oct 30 06:15:21.386715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.386674 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\" (UniqueName: \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\") pod \"81e40a21-5725-4637-bbc6-31bb193af0cf\" (UID: \"81e40a21-5725-4637-bbc6-31bb193af0cf\") " Oct 30 06:15:21.389201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.388963 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca" pod "81e40a21-5725-4637-bbc6-31bb193af0cf" (UID: "81e40a21-5725-4637-bbc6-31bb193af0cf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.390026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.390005 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.390269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.390198 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81e40a21-5725-4637-bbc6-31bb193af0cf-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.390508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.390489 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j75f\" (UniqueName: \"kubernetes.io/projected/f38bd9ab-ce56-41ea-978d-faa7f57f8a09-kube-api-access-2j75f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.390779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.390763 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gp7p\" (UniqueName: \"kubernetes.io/projected/ae2fbf77-9ad7-492d-8908-7dedfa97665c-kube-api-access-9gp7p\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.390869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.390859 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae2fbf77-9ad7-492d-8908-7dedfa97665c-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.394185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.394139 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp" pod "81e40a21-5725-4637-bbc6-31bb193af0cf" (UID: "81e40a21-5725-4637-bbc6-31bb193af0cf"). InnerVolumeSpecName "kube-api-access-87mvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.445177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.445038 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:21.491772 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.491748 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\") pod \"5d0fd932-4293-4670-a1e7-ffea84b24d29\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " Oct 30 06:15:21.492192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.492162 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\" (UniqueName: \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\") pod \"5d0fd932-4293-4670-a1e7-ffea84b24d29\" (UID: \"5d0fd932-4293-4670-a1e7-ffea84b24d29\") " Oct 30 06:15:21.493769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.493743 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mvp\" (UniqueName: \"kubernetes.io/projected/81e40a21-5725-4637-bbc6-31bb193af0cf-kube-api-access-87mvp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.495982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.495943 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh" pod "5d0fd932-4293-4670-a1e7-ffea84b24d29" (UID: "5d0fd932-4293-4670-a1e7-ffea84b24d29"). InnerVolumeSpecName "kube-api-access-rtfvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.522357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.522289 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca" pod "5d0fd932-4293-4670-a1e7-ffea84b24d29" (UID: "5d0fd932-4293-4670-a1e7-ffea84b24d29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.595735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.595702 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d0fd932-4293-4670-a1e7-ffea84b24d29-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.595735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.595733 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtfvh\" (UniqueName: \"kubernetes.io/projected/5d0fd932-4293-4670-a1e7-ffea84b24d29-kube-api-access-rtfvh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.662224 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:21.662191 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec025c0b_1bbe_4792_9b4f_65c55b74d7f7.slice/crio-fe7f9ff96e6db24a297370e1803a051a781d66ca4a634a3d97054f6aefa46556 WatchSource:0}: Error finding container fe7f9ff96e6db24a297370e1803a051a781d66ca4a634a3d97054f6aefa46556: Status 404 returned error can't find the container with id fe7f9ff96e6db24a297370e1803a051a781d66ca4a634a3d97054f6aefa46556 Oct 30 06:15:21.665513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.665475 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:21.665651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.665544 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:21.665651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.665572 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:15:21.676660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.676634 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" path="/var/lib/kubelet/pods/7f801879-e408-4b05-acac-db0dac8e0f41/volumes" Oct 30 06:15:21.677206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.677182 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" path="/var/lib/kubelet/pods/a49655a2-0273-48ad-a6ae-6751667e17ff/volumes" Oct 30 06:15:21.677642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.677628 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" path="/var/lib/kubelet/pods/bbb83f9d-19b3-4a42-947a-081e19c55622/volumes" Oct 30 06:15:21.678406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.678391 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7ab0c7-25ea-42c1-9a86-07b5abc968ce" path="/var/lib/kubelet/pods/ee7ab0c7-25ea-42c1-9a86-07b5abc968ce/volumes" Oct 30 06:15:21.679378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.679359 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:15:21.852479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.852438 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" event={"ID":"ae2fbf77-9ad7-492d-8908-7dedfa97665c","Type":"ContainerDied","Data":"00cf2a24a4303f188789e7508f0bcb800f090b5dbb84cf4347b4caf46ad9e2f5"} Oct 30 06:15:21.852942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.852446 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l" Oct 30 06:15:21.852942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.852567 6171 scope.go:122] "RemoveContainer" containerID="df0837ef65ab72004fa0056a9ae361839ebdc6e65f034df3494248d909429467" Oct 30 06:15:21.855307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.855282 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerStarted","Data":"21b54d99cd867648bd3045310b87b174b0a159c7ad659e9a7f2b8b3ac741f21f"} Oct 30 06:15:21.856928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.856908 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" Oct 30 06:15:21.857078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.856929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww" event={"ID":"5d0fd932-4293-4670-a1e7-ffea84b24d29","Type":"ContainerDied","Data":"981972761e7010661b55e272320cffefb0dd7a9c0189d07527c129f63d5e5fd4"} Oct 30 06:15:21.859275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.859254 6171 generic.go:356] "Generic (PLEG): container finished" podID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerID="a33b8adf873e9fa63dc41529faec3470dcd64d91d1c376047cdf88dc6dfd163f" exitCode=60 Oct 30 06:15:21.859373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.859328 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerDied","Data":"a33b8adf873e9fa63dc41529faec3470dcd64d91d1c376047cdf88dc6dfd163f"} Oct 30 06:15:21.859714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.859696 6171 scope.go:122] "RemoveContainer" containerID="a33b8adf873e9fa63dc41529faec3470dcd64d91d1c376047cdf88dc6dfd163f" Oct 30 06:15:21.860015 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:21.859995 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz_odh-ai-gateway-infra(a11e045d-1571-44f4-843a-67cad6d0d07f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" Oct 30 06:15:21.861594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.861572 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerStarted","Data":"8cd4bbf90d07cf0eb61aaac9501dc15b6aa7bf08b752e406c6ceb45d0c49b17d"} Oct 30 06:15:21.862954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.862935 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" event={"ID":"81e40a21-5725-4637-bbc6-31bb193af0cf","Type":"ContainerDied","Data":"23de0fdfd51e35e050a32e62846aee7f47b03ad7a7ef60a946758336cbec8a19"} Oct 30 06:15:21.863119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.862956 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65" Oct 30 06:15:21.864857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.864833 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" event={"ID":"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7","Type":"ContainerStarted","Data":"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d"} Oct 30 06:15:21.864979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.864875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" event={"ID":"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7","Type":"ContainerStarted","Data":"fe7f9ff96e6db24a297370e1803a051a781d66ca4a634a3d97054f6aefa46556"} Oct 30 06:15:21.866444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.866426 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" Oct 30 06:15:21.866552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.866428 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt" event={"ID":"f38bd9ab-ce56-41ea-978d-faa7f57f8a09","Type":"ContainerDied","Data":"dc3c94c6c7b5e7430c7213f9bf592be075ed5c9a657ed86947b8d0175cdc839c"} Oct 30 06:15:21.866750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.866727 6171 scope.go:122] "RemoveContainer" containerID="d2e84cfaf7239e66e287866205e510a2fa4137a31d8f9d581241377b3e57e835" Oct 30 06:15:21.877868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.877853 6171 scope.go:122] "RemoveContainer" containerID="d0eb063e742a053778a54a10879a166c25b2ce845a0a2c75f4fed549100cb14c" Oct 30 06:15:21.923419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.923390 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:21.929715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.929692 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-a35835-29889015-wcl65"] Oct 30 06:15:21.970023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.969995 6171 scope.go:122] "RemoveContainer" containerID="c77be04f9afe420ee5503c4ff15effb12a14252f5a38764e41db216bd24315b5" Oct 30 06:15:21.974186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.973996 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podStartSLOduration=1.97398234 podStartE2EDuration="1.97398234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:21.96956589 +0000 UTC m=+2580.409943084" watchObservedRunningTime="2026-10-30 06:15:21.97398234 +0000 UTC m=+2580.414359566" Oct 30 06:15:21.984415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:21.984392 6171 scope.go:122] "RemoveContainer" containerID="ba65936d653f2260c09c9034d72b23f4b2383ee0e6eafe49648aeab4792ac103" Oct 30 06:15:22.014750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.014717 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:22.021187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.021164 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-dddc53-29889015-7dcdt"] Oct 30 06:15:22.058104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.058033 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:22.064141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.064117 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-03d349-29889015-4gb9l"] Oct 30 06:15:22.066984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.066966 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:22.069582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.069558 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:22.069688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.069608 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:22.090722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.090693 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:22.199254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.199200 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:22.205013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.204983 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-4a68a1-29889015-7nhww"] Oct 30 06:15:22.909097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.903406 6171 generic.go:356] "Generic (PLEG): container finished" podID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerID="21b54d99cd867648bd3045310b87b174b0a159c7ad659e9a7f2b8b3ac741f21f" exitCode=60 Oct 30 06:15:22.909097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.903522 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerDied","Data":"21b54d99cd867648bd3045310b87b174b0a159c7ad659e9a7f2b8b3ac741f21f"} Oct 30 06:15:22.909097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.903580 6171 scope.go:122] "RemoveContainer" containerID="e2c5c065bd827fc215bb6d950332460897468cbc39af195b0fd1c488950c5087" Oct 30 06:15:22.909097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.904106 6171 scope.go:122] "RemoveContainer" containerID="21b54d99cd867648bd3045310b87b174b0a159c7ad659e9a7f2b8b3ac741f21f" Oct 30 06:15:22.909097 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:22.904603 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg_odh-ai-gateway-infra(1375bbc9-4ccb-4173-bde3-8b9e22f8d731)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" Oct 30 06:15:22.917319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.917288 6171 generic.go:356] "Generic (PLEG): container finished" podID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerID="8cd4bbf90d07cf0eb61aaac9501dc15b6aa7bf08b752e406c6ceb45d0c49b17d" exitCode=60 Oct 30 06:15:22.917436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.917335 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerDied","Data":"8cd4bbf90d07cf0eb61aaac9501dc15b6aa7bf08b752e406c6ceb45d0c49b17d"} Oct 30 06:15:22.917896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.917875 6171 scope.go:122] "RemoveContainer" containerID="8cd4bbf90d07cf0eb61aaac9501dc15b6aa7bf08b752e406c6ceb45d0c49b17d" Oct 30 06:15:22.918347 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:22.918305 6171 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp_odh-ai-gateway-infra(9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" Oct 30 06:15:22.929763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:22.929737 6171 scope.go:122] "RemoveContainer" containerID="f44f27c79666bc1b9e7bf868ad74ea87fc80c74b4868bc60a38891931f5878d1" Oct 30 06:15:23.068352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.068307 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:23.068624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.068595 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:23.129111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.128501 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:23.173016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.172988 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:23.205436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.205401 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:23.335070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.330653 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\" (UniqueName: \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\") pod \"a11e045d-1571-44f4-843a-67cad6d0d07f\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " Oct 30 06:15:23.335070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.330861 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\") pod \"a11e045d-1571-44f4-843a-67cad6d0d07f\" (UID: \"a11e045d-1571-44f4-843a-67cad6d0d07f\") " Oct 30 06:15:23.360298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.360263 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca" pod "a11e045d-1571-44f4-843a-67cad6d0d07f" (UID: "a11e045d-1571-44f4-843a-67cad6d0d07f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:23.363316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.363280 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd" pod "a11e045d-1571-44f4-843a-67cad6d0d07f" (UID: "a11e045d-1571-44f4-843a-67cad6d0d07f"). InnerVolumeSpecName "kube-api-access-4mpvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:23.432832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.432754 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mpvd\" (UniqueName: \"kubernetes.io/projected/a11e045d-1571-44f4-843a-67cad6d0d07f-kube-api-access-4mpvd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.432832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.432785 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a11e045d-1571-44f4-843a-67cad6d0d07f-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.677713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.677678 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" path="/var/lib/kubelet/pods/5d0fd932-4293-4670-a1e7-ffea84b24d29/volumes" Oct 30 06:15:23.678286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.678267 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" path="/var/lib/kubelet/pods/81e40a21-5725-4637-bbc6-31bb193af0cf/volumes" Oct 30 06:15:23.678687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.678674 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" path="/var/lib/kubelet/pods/ae2fbf77-9ad7-492d-8908-7dedfa97665c/volumes" Oct 30 06:15:23.679416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.679401 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" path="/var/lib/kubelet/pods/f38bd9ab-ce56-41ea-978d-faa7f57f8a09/volumes" Oct 30 06:15:23.933691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.933671 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" Oct 30 06:15:23.934088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.933664 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz" event={"ID":"a11e045d-1571-44f4-843a-67cad6d0d07f","Type":"ContainerDied","Data":"80af537b5fca284d8125b2f5eba0b7d095c3c66f95a496f95d13a27e4842eade"} Oct 30 06:15:23.934088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:23.933827 6171 scope.go:122] "RemoveContainer" containerID="a33b8adf873e9fa63dc41529faec3470dcd64d91d1c376047cdf88dc6dfd163f" Oct 30 06:15:24.067426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.067395 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:24.067583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.067443 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:24.110067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.109592 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:24.119079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.119025 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-c78066-29889015-hrtdz"] Oct 30 06:15:24.179295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.179273 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:24.248441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.248416 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:24.249285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.249260 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\") pod \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " Oct 30 06:15:24.249469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.249451 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\" (UniqueName: \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\") pod \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\" (UID: \"1375bbc9-4ccb-4173-bde3-8b9e22f8d731\") " Oct 30 06:15:24.251264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.250594 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca" pod "1375bbc9-4ccb-4173-bde3-8b9e22f8d731" (UID: "1375bbc9-4ccb-4173-bde3-8b9e22f8d731"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.252370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.252343 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4" pod "1375bbc9-4ccb-4173-bde3-8b9e22f8d731" (UID: "1375bbc9-4ccb-4173-bde3-8b9e22f8d731"). InnerVolumeSpecName "kube-api-access-78rs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.351363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.351196 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\") pod \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " Oct 30 06:15:24.351548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.351493 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\" (UniqueName: \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\") pod \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\" (UID: \"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66\") " Oct 30 06:15:24.352363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.352328 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78rs4\" (UniqueName: \"kubernetes.io/projected/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-kube-api-access-78rs4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.352363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.352366 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1375bbc9-4ccb-4173-bde3-8b9e22f8d731-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.352579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.352533 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca" pod "9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" (UID: "9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.354998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.354975 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h" pod "9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" (UID: "9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66"). InnerVolumeSpecName "kube-api-access-rpj6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.453709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.453670 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-maas-api-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.453709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.453714 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpj6h\" (UniqueName: \"kubernetes.io/projected/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66-kube-api-access-rpj6h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.947823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.947794 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" Oct 30 06:15:24.948313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.947839 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp" event={"ID":"9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66","Type":"ContainerDied","Data":"6d188d6cef0dd288fc805b065ac48775af9db827cf3b38431981401facd0bb1c"} Oct 30 06:15:24.948313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.947898 6171 scope.go:122] "RemoveContainer" containerID="8cd4bbf90d07cf0eb61aaac9501dc15b6aa7bf08b752e406c6ceb45d0c49b17d" Oct 30 06:15:24.949699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.949660 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" Oct 30 06:15:24.949844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.949660 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg" event={"ID":"1375bbc9-4ccb-4173-bde3-8b9e22f8d731","Type":"ContainerDied","Data":"46cb5d2877f805a41089890a9c878ae217b650ff0853dcf8a0eef4504d4ecca2"} Oct 30 06:15:24.962278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:24.962263 6171 scope.go:122] "RemoveContainer" containerID="21b54d99cd867648bd3045310b87b174b0a159c7ad659e9a7f2b8b3ac741f21f" Oct 30 06:15:25.075071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.073976 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:25.079070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.075681 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:25.108067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.105113 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:25.108067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.105189 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-24db22-29889015-7m5hg"] Oct 30 06:15:25.131067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.130096 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:25.136885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.136810 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-477f6b-29889015-mpfkp"] Oct 30 06:15:25.676557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.676523 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" path="/var/lib/kubelet/pods/1375bbc9-4ccb-4173-bde3-8b9e22f8d731/volumes" Oct 30 06:15:25.677133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.677110 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" path="/var/lib/kubelet/pods/9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66/volumes" Oct 30 06:15:25.677764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:25.677744 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" path="/var/lib/kubelet/pods/a11e045d-1571-44f4-843a-67cad6d0d07f/volumes" Oct 30 06:15:26.067347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:26.067318 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:26.067768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:26.067385 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:26.416871 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:26.416781 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:15:27.067540 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:27.067506 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:27.068012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:27.067566 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:28.067308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.067268 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" start-of-body= Oct 30 06:15:28.067501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.067333 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.101:15021/healthz/ready\": dial tcp 10.133.1.101:15021: connect: connection refused" Oct 30 06:15:28.537980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.537945 6171 scope.go:122] "RemoveContainer" containerID="93de8a5ef3fd79be11bc91530fa9fc20941469d18f1d085771830ad95708295d" Oct 30 06:15:28.563114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.563061 6171 scope.go:122] "RemoveContainer" containerID="a6804f8759ac0a1bdd6ed185949e6363a256e7eaf440104c8f3185f170557297" Oct 30 06:15:28.575228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.575207 6171 scope.go:122] "RemoveContainer" containerID="0187e5cca985f92e5f49c3fb02143f1356d7a3349d3e7b80d5157e253997dc4c" Oct 30 06:15:28.586153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:28.586137 6171 scope.go:122] "RemoveContainer" containerID="9954cb9b3d2ec76bcca5ed6189aab746819a59e5e6c9d14d41cd9ea2b550c9cf" Oct 30 06:15:29.127780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:29.127748 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:29.129895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:29.129635 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:29.132321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:29.132277 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:15:31.935062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.935019 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:15:31.936148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936124 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.936148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936146 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936168 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936174 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936187 6171 cpu_manager.go:414] "containerMap: removing container" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936195 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936212 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936218 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936231 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936238 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936257 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936264 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936291 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936297 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f801879-e408-4b05-acac-db0dac8e0f41" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936309 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936316 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936328 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936343 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936358 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936364 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:31.936378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936382 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936389 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936398 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936405 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936420 6171 cpu_manager.go:414] "containerMap: removing container" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936427 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936436 6171 cpu_manager.go:414] "containerMap: removing container" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936444 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936459 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936466 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936477 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936484 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936497 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936503 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936524 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936531 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936550 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936556 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936568 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936574 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936584 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936590 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936797 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936812 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936824 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936835 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936845 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936856 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936870 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936880 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbb83f9d-19b3-4a42-947a-081e19c55622" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936892 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936901 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936913 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a49655a2-0273-48ad-a6ae-6751667e17ff" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936925 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936935 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936944 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936954 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936964 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936973 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2fbf77-9ad7-492d-8908-7dedfa97665c" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936985 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.936993 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:31.937373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.937002 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:31.945279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.945259 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:31.963632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.963598 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:15:31.970093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:31.970058 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-bf87e1-serving-cert\"" Oct 30 06:15:32.013388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.013359 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.013541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.013502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6hbg\" (UniqueName: \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.013678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.013659 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.013755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.013707 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.119080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.118471 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.119080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.118598 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6hbg\" (UniqueName: \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.119080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.118768 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.119080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.118811 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.133084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.120160 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.133084 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:32.120343 6171 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-serving-cert: secret "maas-api-e2e-ipp-a-bf87e1-serving-cert" not found Oct 30 06:15:32.133084 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:32.120404 6171 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls podName:6143e0ba-e5c5-41d5-9937-53e7a706cd55 nodeName:}" failed. No retries permitted until 2026-10-30 06:15:32.620385566 +0000 UTC m=+2591.060762744 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls") pod "maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" (UID: "6143e0ba-e5c5-41d5-9937-53e7a706cd55") : secret "maas-api-e2e-ipp-a-bf87e1-serving-cert" not found Oct 30 06:15:32.133084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.125786 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.166179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.162692 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6hbg\" (UniqueName: \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.625817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.625766 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.628580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.628549 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") pod \"maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:32.864118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:32.864086 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:33.275080 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:33.271163 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6143e0ba_e5c5_41d5_9937_53e7a706cd55.slice/crio-5ad5b91a75ac9746d65b2035562bb2b4200eed61c285409245bd63a9795644eb WatchSource:0}: Error finding container 5ad5b91a75ac9746d65b2035562bb2b4200eed61c285409245bd63a9795644eb: Status 404 returned error can't find the container with id 5ad5b91a75ac9746d65b2035562bb2b4200eed61c285409245bd63a9795644eb Oct 30 06:15:33.275080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:33.272291 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:15:33.298194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:33.298143 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:15:34.051849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:34.051808 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" event={"ID":"6143e0ba-e5c5-41d5-9937-53e7a706cd55","Type":"ContainerStarted","Data":"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f"} Oct 30 06:15:34.052031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:34.051904 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" event={"ID":"6143e0ba-e5c5-41d5-9937-53e7a706cd55","Type":"ContainerStarted","Data":"5ad5b91a75ac9746d65b2035562bb2b4200eed61c285409245bd63a9795644eb"} Oct 30 06:15:34.052031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:34.051965 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:34.100958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:34.100900 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" podStartSLOduration=2.7306056119999997 podStartE2EDuration="3.100885793s" totalImagesPullingTime="370.280181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:34.099238348 +0000 UTC m=+2592.539615543" watchObservedRunningTime="2026-10-30 06:15:34.100885793 +0000 UTC m=+2592.541262985" Oct 30 06:15:40.109068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:40.103386 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.730110 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731701 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731722 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38bd9ab-ce56-41ea-978d-faa7f57f8a09" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731803 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731812 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d0fd932-4293-4670-a1e7-ffea84b24d29" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731855 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.731863 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a11e045d-1571-44f4-843a-67cad6d0d07f" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.732122 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1375bbc9-4ccb-4173-bde3-8b9e22f8d731" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.732138 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e40a21-5725-4637-bbc6-31bb193af0cf" containerName="cleanup" Oct 30 06:15:48.737067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.732150 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d8eb99f-1e3e-4ec8-afd0-f77a83adfa66" containerName="cleanup" Oct 30 06:15:48.742066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.740331 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.775069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.771574 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:15:48.803084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.800361 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dddc53\"" Oct 30 06:15:48.803084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.800475 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dddc53-dockercfg-kg42j\"" Oct 30 06:15:48.827070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.821293 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slbdz\" (UniqueName: \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.827070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.821665 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.927108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.906421 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:15:48.927108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.915846 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:48.927108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.924924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.927108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.925110 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slbdz\" (UniqueName: \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.927108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.926121 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:48.933353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.915753 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:15:48.966090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:48.964700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slbdz\" (UniqueName: \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\") pod \"payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:49.028126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.027282 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd2x\" (UniqueName: \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.028126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.027407 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.106274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.106246 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:49.129392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.129365 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd2x\" (UniqueName: \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.129525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.129418 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.130204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.130182 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.158438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.158406 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd2x\" (UniqueName: \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\") pod \"payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.238728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.238694 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:49.450128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.450093 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp"] Oct 30 06:15:49.482244 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:49.480732 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea40ed47_98ce_47c7_b0d8_dacb65959077.slice/crio-52dcacbf773f530e527b1464d523124e42988e3471c2681acaab3e562ac0961f WatchSource:0}: Error finding container 52dcacbf773f530e527b1464d523124e42988e3471c2681acaab3e562ac0961f: Status 404 returned error can't find the container with id 52dcacbf773f530e527b1464d523124e42988e3471c2681acaab3e562ac0961f Oct 30 06:15:49.492933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.492906 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.510214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.509111 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp"] Oct 30 06:15:49.512948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.512921 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:15:49.513086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.512944 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:15:49.514978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.514959 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:15:49.539286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.539256 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.539408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.539371 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xmmdk\" (UniqueName: \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.539461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.539435 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.573313 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:49.573288 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bddcaa_d7e8_478d_a87f_89998720be22.slice/crio-a63a8dcf299cf9ba60e9c72d81ab568a668c6cf5ea9668640e909bb2b985cc62 WatchSource:0}: Error finding container a63a8dcf299cf9ba60e9c72d81ab568a668c6cf5ea9668640e909bb2b985cc62: Status 404 returned error can't find the container with id a63a8dcf299cf9ba60e9c72d81ab568a668c6cf5ea9668640e909bb2b985cc62 Oct 30 06:15:49.574486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.574461 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:15:49.641411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.641384 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.641516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.641465 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xmmdk\" (UniqueName: \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.641562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.641521 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.642150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.642131 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.643972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.643951 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.666879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.666859 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xmmdk\" (UniqueName: \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\") pod \"maas-api-revoke-sub-83ed428d-7mlcp\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:49.818573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:49.818542 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:50.216090 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:15:50.210153 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a121e4_e83b_42f6_b368_8882fb8f6b21.slice/crio-3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0 WatchSource:0}: Error finding container 3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0: Status 404 returned error can't find the container with id 3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0 Oct 30 06:15:50.231459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:50.231428 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp"] Oct 30 06:15:50.366634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:50.366588 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" event={"ID":"69bddcaa-d7e8-478d-a87f-89998720be22","Type":"ContainerStarted","Data":"a63a8dcf299cf9ba60e9c72d81ab568a668c6cf5ea9668640e909bb2b985cc62"} Oct 30 06:15:50.378159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:50.375253 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" event={"ID":"78a121e4-e83b-42f6-b368-8882fb8f6b21","Type":"ContainerStarted","Data":"3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0"} Oct 30 06:15:50.380328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:50.380292 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" event={"ID":"ea40ed47-98ce-47c7-b0d8-dacb65959077","Type":"ContainerStarted","Data":"52dcacbf773f530e527b1464d523124e42988e3471c2681acaab3e562ac0961f"} Oct 30 06:15:51.395742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:51.395698 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" event={"ID":"78a121e4-e83b-42f6-b368-8882fb8f6b21","Type":"ContainerStarted","Data":"27acc75bdde1b3084ef57dd1fd66b0001b45984954a6d7d1e80e5c6e950d8fa3"} Oct 30 06:15:51.440962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:51.440910 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" podStartSLOduration=2.440896623 podStartE2EDuration="2.440896623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:51.440232766 +0000 UTC m=+2609.880609960" watchObservedRunningTime="2026-10-30 06:15:51.440896623 +0000 UTC m=+2609.881273809" Oct 30 06:15:52.407815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.407772 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" event={"ID":"69bddcaa-d7e8-478d-a87f-89998720be22","Type":"ContainerStarted","Data":"0cbb7e27363d4e451924651baaf39a94ba6217a86948c57a7d2af385de05af61"} Oct 30 06:15:52.408290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.407866 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:15:52.410177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.410151 6171 generic.go:356] "Generic (PLEG): container finished" podID="78a121e4-e83b-42f6-b368-8882fb8f6b21" containerID="27acc75bdde1b3084ef57dd1fd66b0001b45984954a6d7d1e80e5c6e950d8fa3" exitCode=0 Oct 30 06:15:52.410327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.410239 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" event={"ID":"78a121e4-e83b-42f6-b368-8882fb8f6b21","Type":"ContainerDied","Data":"27acc75bdde1b3084ef57dd1fd66b0001b45984954a6d7d1e80e5c6e950d8fa3"} Oct 30 06:15:52.412221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.412192 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" event={"ID":"ea40ed47-98ce-47c7-b0d8-dacb65959077","Type":"ContainerStarted","Data":"8da316078834fc246001ac3a0c31e5105624f116aa6e0773fcc0eefe6a59e694"} Oct 30 06:15:52.412329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.412295 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:15:52.463009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.462959 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" podStartSLOduration=2.09160388 podStartE2EDuration="4.462941582s" totalImagesPullingTime="2.371337702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:52.454222154 +0000 UTC m=+2610.894599346" watchObservedRunningTime="2026-10-30 06:15:52.462941582 +0000 UTC m=+2610.903318780" Oct 30 06:15:52.548211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:52.548154 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" podStartSLOduration=2.079199121 podStartE2EDuration="4.548140938s" totalImagesPullingTime="2.468941817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:52.543297415 +0000 UTC m=+2610.983674611" watchObservedRunningTime="2026-10-30 06:15:52.548140938 +0000 UTC m=+2610.988518128" Oct 30 06:15:53.622217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.622190 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:53.644423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.644398 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\" (UniqueName: \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\") pod \"78a121e4-e83b-42f6-b368-8882fb8f6b21\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " Oct 30 06:15:53.644566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.644468 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\") pod \"78a121e4-e83b-42f6-b368-8882fb8f6b21\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " Oct 30 06:15:53.644706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.644688 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\") pod \"78a121e4-e83b-42f6-b368-8882fb8f6b21\" (UID: \"78a121e4-e83b-42f6-b368-8882fb8f6b21\") " Oct 30 06:15:53.645060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.645018 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca" pod "78a121e4-e83b-42f6-b368-8882fb8f6b21" (UID: "78a121e4-e83b-42f6-b368-8882fb8f6b21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:53.645324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.645303 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:53.647117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.647091 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca" pod "78a121e4-e83b-42f6-b368-8882fb8f6b21" (UID: "78a121e4-e83b-42f6-b368-8882fb8f6b21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:15:53.647222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.647160 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk" pod "78a121e4-e83b-42f6-b368-8882fb8f6b21" (UID: "78a121e4-e83b-42f6-b368-8882fb8f6b21"). InnerVolumeSpecName "kube-api-access-xmmdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:53.746835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.746799 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xmmdk\" (UniqueName: \"kubernetes.io/projected/78a121e4-e83b-42f6-b368-8882fb8f6b21-kube-api-access-xmmdk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:53.746835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:53.746830 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78a121e4-e83b-42f6-b368-8882fb8f6b21-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:15:54.434179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:54.434147 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" Oct 30 06:15:54.434179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:54.434168 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp" event={"ID":"78a121e4-e83b-42f6-b368-8882fb8f6b21","Type":"ContainerDied","Data":"3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0"} Oct 30 06:15:54.434392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:54.434221 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3027e4c7dea96147c09b019b268957f5b17268b54a90d860858d75aba9a9ffa0" Oct 30 06:15:56.384471 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:15:56.384432 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:15:59.850351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.850320 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp"] Oct 30 06:15:59.851231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.851216 6171 cpu_manager.go:414] "containerMap: removing container" podUID="78a121e4-e83b-42f6-b368-8882fb8f6b21" containerName="revoke-keys" Oct 30 06:15:59.851294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.851233 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78a121e4-e83b-42f6-b368-8882fb8f6b21" containerName="revoke-keys" Oct 30 06:15:59.851388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.851378 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="78a121e4-e83b-42f6-b368-8882fb8f6b21" containerName="revoke-keys" Oct 30 06:15:59.855681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.855661 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:15:59.873347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.872823 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:15:59.873347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.873003 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:15:59.876619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.876593 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp"] Oct 30 06:15:59.955589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.955555 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:15:59.955813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.955791 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:15:59.955878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:15:59.955852 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbxc\" (UniqueName: \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.058808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.058768 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.059001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.058863 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbxc\" (UniqueName: \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.059103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.059074 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.059529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.059509 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.061843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.061816 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.084846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.084815 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbxc\" (UniqueName: \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\") pod \"maas-api-revoke-sub-e0d15423-7sqkp\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.171876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.171785 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:00.500210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.500178 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp"] Oct 30 06:16:00.501192 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:00.501164 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74d26f15_aa50_418c_95ea_4fa92d9eaf6f.slice/crio-d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162 WatchSource:0}: Error finding container d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162: Status 404 returned error can't find the container with id d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162 Oct 30 06:16:00.531604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:00.531571 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" event={"ID":"74d26f15-aa50-418c-95ea-4fa92d9eaf6f","Type":"ContainerStarted","Data":"d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162"} Oct 30 06:16:01.543715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:01.543679 6171 generic.go:356] "Generic (PLEG): container finished" podID="74d26f15-aa50-418c-95ea-4fa92d9eaf6f" containerID="218f508c80fa86a433adc0a7fd3a590ee4d15e8618446e0a1fd2f475e9652e40" exitCode=0 Oct 30 06:16:01.544059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:01.543759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" event={"ID":"74d26f15-aa50-418c-95ea-4fa92d9eaf6f","Type":"ContainerDied","Data":"218f508c80fa86a433adc0a7fd3a590ee4d15e8618446e0a1fd2f475e9652e40"} Oct 30 06:16:02.737419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.737387 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:02.809681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.809589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\") pod \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " Oct 30 06:16:02.809834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.809803 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\" (UniqueName: \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\") pod \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " Oct 30 06:16:02.809885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.809870 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\") pod \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\" (UID: \"74d26f15-aa50-418c-95ea-4fa92d9eaf6f\") " Oct 30 06:16:02.809996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.809949 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca" pod "74d26f15-aa50-418c-95ea-4fa92d9eaf6f" (UID: "74d26f15-aa50-418c-95ea-4fa92d9eaf6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:02.810762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.810732 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:02.812598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.812568 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca" pod "74d26f15-aa50-418c-95ea-4fa92d9eaf6f" (UID: "74d26f15-aa50-418c-95ea-4fa92d9eaf6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:02.812740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.812683 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc" pod "74d26f15-aa50-418c-95ea-4fa92d9eaf6f" (UID: "74d26f15-aa50-418c-95ea-4fa92d9eaf6f"). InnerVolumeSpecName "kube-api-access-mpbxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:02.916623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.915326 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbxc\" (UniqueName: \"kubernetes.io/projected/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-kube-api-access-mpbxc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:02.916854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:02.916835 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74d26f15-aa50-418c-95ea-4fa92d9eaf6f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:03.163240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.163147 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:16:03.459118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.453456 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:16:03.459118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.454268 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:16:03.570290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.570248 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" event={"ID":"74d26f15-aa50-418c-95ea-4fa92d9eaf6f","Type":"ContainerDied","Data":"d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162"} Oct 30 06:16:03.570290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.570274 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp" Oct 30 06:16:03.570545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:03.570316 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d753801aaebfdf4fca8d1b480aa53afe528bf1d73b4b19deaaf0139191bd9162" Oct 30 06:16:04.075420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:04.075385 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7"] Oct 30 06:16:04.081552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:04.081524 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5a6380f-9shx7"] Oct 30 06:16:05.676167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:05.676137 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5735c8ac-caa5-40b6-b466-499440aefdb8" path="/var/lib/kubelet/pods/5735c8ac-caa5-40b6-b466-499440aefdb8/volumes" Oct 30 06:16:08.164416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.164377 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" podUID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" containerName="payload-processing" containerID="cri-o://4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245" gracePeriod=25 Oct 30 06:16:08.461341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.461312 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:16:08.565839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.565782 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\" (UniqueName: \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\") pod \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " Oct 30 06:16:08.566020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.565855 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\") pod \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\" (UID: \"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9\") " Oct 30 06:16:08.566311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.566273 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume" pod "aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" (UID: "aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:08.566793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.566770 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.568663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.568637 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl" pod "aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" (UID: "aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9"). InnerVolumeSpecName "kube-api-access-wxpsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:08.710662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.691803 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxpsl\" (UniqueName: \"kubernetes.io/projected/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9-kube-api-access-wxpsl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.721178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.721146 6171 generic.go:356] "Generic (PLEG): container finished" podID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" containerID="4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245" exitCode=0 Oct 30 06:16:08.721355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.721207 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" Oct 30 06:16:08.721355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.721237 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" event={"ID":"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9","Type":"ContainerDied","Data":"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245"} Oct 30 06:16:08.721355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.721295 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr" event={"ID":"aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9","Type":"ContainerDied","Data":"9bb4feea0f6bfca1d9e18de4749206331ee2b4e26f3fdfdd4ba5b69b23e08c40"} Oct 30 06:16:08.721355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.721322 6171 scope.go:122] "RemoveContainer" containerID="4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245" Oct 30 06:16:08.738339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.738312 6171 scope.go:122] "RemoveContainer" containerID="4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245" Oct 30 06:16:08.738794 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:08.738773 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245\": container with ID starting with 4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245 not found: ID does not exist" containerID="4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245" Oct 30 06:16:08.738918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.738800 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245"} err="failed to get container status \"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245\": rpc error: code = NotFound desc = could not find container \"4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245\": container with ID starting with 4c7993d75ac79a85a6c22b8725b85d53ea7244843357e6b2447a46748db82245 not found: ID does not exist" Oct 30 06:16:08.783675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.783654 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:16:08.789152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:08.789133 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-5467db566c-sffpr"] Oct 30 06:16:09.680020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:09.679980 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" path="/var/lib/kubelet/pods/aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9/volumes" Oct 30 06:16:11.838541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.838506 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:16:11.839428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839412 6171 cpu_manager.go:414] "containerMap: removing container" podUID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" containerName="payload-processing" Oct 30 06:16:11.839498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839429 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" containerName="payload-processing" Oct 30 06:16:11.839498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839442 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74d26f15-aa50-418c-95ea-4fa92d9eaf6f" containerName="revoke-keys" Oct 30 06:16:11.839498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839448 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74d26f15-aa50-418c-95ea-4fa92d9eaf6f" containerName="revoke-keys" Oct 30 06:16:11.839605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839576 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed476ee-3ad0-4c21-b3c7-6b122fb6ceb9" containerName="payload-processing" Oct 30 06:16:11.839605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.839586 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="74d26f15-aa50-418c-95ea-4fa92d9eaf6f" containerName="revoke-keys" Oct 30 06:16:11.845039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.845022 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:11.875391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.875367 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f7ad21-dockercfg-vrs6k\"" Oct 30 06:16:11.895181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.895160 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f7ad21\"" Oct 30 06:16:11.903426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.903406 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:16:11.982070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.975770 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb2nl\" (UniqueName: \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:11.982070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:11.975916 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.093749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.093662 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:16:12.132986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.119358 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb2nl\" (UniqueName: \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.132986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.119503 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.132986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.120721 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.242754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.242636 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.249888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.249819 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:16:12.253338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.253293 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb2nl\" (UniqueName: \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.345506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.345438 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.345506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.345492 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2fg2\" (UniqueName: \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.448117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.448071 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2fg2\" (UniqueName: \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.448436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.448420 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.449165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.449138 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.454279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.454258 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:16:12.460125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.460103 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:12.475103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.475077 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2fg2\" (UniqueName: \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\") pod \"payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.641941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.641704 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:12.980713 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:12.980681 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58567eaf_73da_45e5_9c46_1a2079d8aa0f.slice/crio-3525698d79c375521b492f7b31f215331505cf77060bbdcdf182a5e21a76967c WatchSource:0}: Error finding container 3525698d79c375521b492f7b31f215331505cf77060bbdcdf182a5e21a76967c: Status 404 returned error can't find the container with id 3525698d79c375521b492f7b31f215331505cf77060bbdcdf182a5e21a76967c Oct 30 06:16:12.982328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:12.982271 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:16:13.069299 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:13.069272 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4095fa43_781a_47a2_b7db_b39205b0a9b3.slice/crio-3ff76c5ad2f6366ff878357b2b903f094c6a287ec187a4f727f7637829e17b82 WatchSource:0}: Error finding container 3ff76c5ad2f6366ff878357b2b903f094c6a287ec187a4f727f7637829e17b82: Status 404 returned error can't find the container with id 3ff76c5ad2f6366ff878357b2b903f094c6a287ec187a4f727f7637829e17b82 Oct 30 06:16:13.071732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.071652 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:16:13.277385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.277288 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:16:13.763539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.763502 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:16:13.768712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.768691 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.787643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.787609 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" event={"ID":"58567eaf-73da-45e5-9c46-1a2079d8aa0f","Type":"ContainerStarted","Data":"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58"} Oct 30 06:16:13.787788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.787667 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" event={"ID":"58567eaf-73da-45e5-9c46-1a2079d8aa0f","Type":"ContainerStarted","Data":"3525698d79c375521b492f7b31f215331505cf77060bbdcdf182a5e21a76967c"} Oct 30 06:16:13.788262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.788230 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-91f42c-openshift-default-dockercfg-5blpg\"" Oct 30 06:16:13.788456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.788431 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:13.790034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.790002 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" event={"ID":"4095fa43-781a-47a2-b7db-b39205b0a9b3","Type":"ContainerStarted","Data":"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad"} Oct 30 06:16:13.790186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.790072 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" event={"ID":"4095fa43-781a-47a2-b7db-b39205b0a9b3","Type":"ContainerStarted","Data":"3ff76c5ad2f6366ff878357b2b903f094c6a287ec187a4f727f7637829e17b82"} Oct 30 06:16:13.790746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.790724 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:13.794927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.794899 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:16:13.843357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843325 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.843508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843400 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.843508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843435 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.843711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843684 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.843849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843830 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.844010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.843991 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljdt\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.844081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.844064 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.844140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.844101 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.844140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.844127 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.869966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.869923 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" podStartSLOduration=2.869911673 podStartE2EDuration="2.869911673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:13.865321497 +0000 UTC m=+2632.305698690" watchObservedRunningTime="2026-10-30 06:16:13.869911673 +0000 UTC m=+2632.310288865" Oct 30 06:16:13.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.911398 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" podStartSLOduration=2.911386835 podStartE2EDuration="2.911386835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:13.908947282 +0000 UTC m=+2632.349324475" watchObservedRunningTime="2026-10-30 06:16:13.911386835 +0000 UTC m=+2632.351764066" Oct 30 06:16:13.945195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945170 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945245 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945348 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljdt\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945380 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945406 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945526 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945627 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945691 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945790 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.945975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945967 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.946129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.945993 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.946226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.946205 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.947862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.947841 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.948431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.948412 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.970661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.970635 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:13.970773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:13.970722 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljdt\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\") pod \"e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:14.086862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.086465 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:14.428634 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:14.428606 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ffc362_a4ce_4f99_b0c3_a41834795deb.slice/crio-b307b2b84a5adb4248312debdcdad3174465bffc7aedce7f9acd69598f8f1cc5 WatchSource:0}: Error finding container b307b2b84a5adb4248312debdcdad3174465bffc7aedce7f9acd69598f8f1cc5: Status 404 returned error can't find the container with id b307b2b84a5adb4248312debdcdad3174465bffc7aedce7f9acd69598f8f1cc5 Oct 30 06:16:14.430140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.430115 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:16:14.431815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.431780 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:14.431904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.431846 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:14.431904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.431877 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:14.803515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.803482 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" event={"ID":"00ffc362-a4ce-4f99-b0c3-a41834795deb","Type":"ContainerStarted","Data":"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2"} Oct 30 06:16:14.803723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.803537 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" event={"ID":"00ffc362-a4ce-4f99-b0c3-a41834795deb","Type":"ContainerStarted","Data":"b307b2b84a5adb4248312debdcdad3174465bffc7aedce7f9acd69598f8f1cc5"} Oct 30 06:16:14.855085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:14.854998 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podStartSLOduration=1.854984855 podStartE2EDuration="1.854984855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:14.854504737 +0000 UTC m=+2633.294881930" watchObservedRunningTime="2026-10-30 06:16:14.854984855 +0000 UTC m=+2633.295362047" Oct 30 06:16:15.089068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:15.088969 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:15.090219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:15.090196 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:15.090278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:15.090243 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:16.036791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:16.036755 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l"] Oct 30 06:16:16.048973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:16.048942 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-74d02af5-7ch6l"] Oct 30 06:16:16.105338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:16.105303 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:16.105943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:16.105912 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:16.516070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:16.515088 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:16:17.102070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:17.098418 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:17.102070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:17.098478 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:17.701072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:17.697215 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a59416ab-f8c7-499c-87bf-857da99648fd" path="/var/lib/kubelet/pods/a59416ab-f8c7-499c-87bf-857da99648fd/volumes" Oct 30 06:16:18.102072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:18.091352 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:18.102072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:18.091411 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:19.094070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:19.088526 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:19.094070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:19.088586 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:20.120232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:20.120187 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:20.120992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:20.120955 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:21.126071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.107144 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:21.126071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.107207 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:21.366234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.361184 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5"] Oct 30 06:16:21.470189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.469283 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5"] Oct 30 06:16:21.470189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.469424 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.551267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.551232 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:16:21.552223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.552198 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:21.576096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.576061 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbhbs\" (UniqueName: \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.576374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.576347 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.576703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.576678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.740555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.740259 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.742119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.740832 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbhbs\" (UniqueName: \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.742119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.741062 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.744488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.744462 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.764286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.764256 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.773844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.773814 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbhbs\" (UniqueName: \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\") pod \"maas-api-revoke-sub-e4187f4a-fsxs5\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:21.907593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.907561 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:16:21.912066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:21.910382 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:22.107066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:22.106846 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" start-of-body= Oct 30 06:16:22.107066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:22.106906 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.109:15021/healthz/ready\": dial tcp 10.133.1.109:15021: connect: connection refused" Oct 30 06:16:22.653271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:22.653240 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5"] Oct 30 06:16:23.067810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:23.067782 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" event={"ID":"9f59fe6b-a375-46a5-bc76-8185caf669be","Type":"ContainerStarted","Data":"1669a2dcfdc1e202ed1bc2c966be452ee6ab820039c0386a02919c98a1bfd8a1"} Oct 30 06:16:23.112407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:23.112371 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:23.114448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:23.114425 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:23.134180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:23.126381 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:16:24.046100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:24.046031 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd"] Oct 30 06:16:24.052487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:24.052463 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c610016d-vmpjd"] Oct 30 06:16:24.080005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:24.079975 6171 generic.go:356] "Generic (PLEG): container finished" podID="9f59fe6b-a375-46a5-bc76-8185caf669be" containerID="c8f44ec3c1b840ff1220d3af550352ffd2e158438aca23d679f44b4ef47ee8ba" exitCode=0 Oct 30 06:16:24.080165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:24.080086 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" event={"ID":"9f59fe6b-a375-46a5-bc76-8185caf669be","Type":"ContainerDied","Data":"c8f44ec3c1b840ff1220d3af550352ffd2e158438aca23d679f44b4ef47ee8ba"} Oct 30 06:16:24.638925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:24.638883 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:16:25.745149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.745112 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0917f3-cd24-43f9-b95f-4b8149066f68" path="/var/lib/kubelet/pods/1a0917f3-cd24-43f9-b95f-4b8149066f68/volumes" Oct 30 06:16:25.820798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.820766 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:16:25.827089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.827011 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:16:25.833959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.833930 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:25.914249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.914103 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\") pod \"9f59fe6b-a375-46a5-bc76-8185caf669be\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " Oct 30 06:16:25.916143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.914589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\" (UniqueName: \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\") pod \"9f59fe6b-a375-46a5-bc76-8185caf669be\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " Oct 30 06:16:25.916143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.914628 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\") pod \"9f59fe6b-a375-46a5-bc76-8185caf669be\" (UID: \"9f59fe6b-a375-46a5-bc76-8185caf669be\") " Oct 30 06:16:25.916659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.916465 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca" pod "9f59fe6b-a375-46a5-bc76-8185caf669be" (UID: "9f59fe6b-a375-46a5-bc76-8185caf669be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:25.917941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.917919 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca" pod "9f59fe6b-a375-46a5-bc76-8185caf669be" (UID: "9f59fe6b-a375-46a5-bc76-8185caf669be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:25.920055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:25.920012 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs" pod "9f59fe6b-a375-46a5-bc76-8185caf669be" (UID: "9f59fe6b-a375-46a5-bc76-8185caf669be"). InnerVolumeSpecName "kube-api-access-cbhbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:26.016669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.016635 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.016669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.016667 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbhbs\" (UniqueName: \"kubernetes.io/projected/9f59fe6b-a375-46a5-bc76-8185caf669be-kube-api-access-cbhbs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.016669 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.016677 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f59fe6b-a375-46a5-bc76-8185caf669be-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.182056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.182009 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" Oct 30 06:16:26.182272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.182008 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5" event={"ID":"9f59fe6b-a375-46a5-bc76-8185caf669be","Type":"ContainerDied","Data":"1669a2dcfdc1e202ed1bc2c966be452ee6ab820039c0386a02919c98a1bfd8a1"} Oct 30 06:16:26.182272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:26.182184 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1669a2dcfdc1e202ed1bc2c966be452ee6ab820039c0386a02919c98a1bfd8a1" Oct 30 06:16:26.382122 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:26.382088 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:16:27.477527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.477475 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" podUID="e1f93a83-d39a-4070-8c39-94e94afe1d31" containerName="main" containerID="cri-o://c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02" gracePeriod=45 Oct 30 06:16:27.869575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.869236 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:16:27.965481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.965668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965490 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.965668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965599 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\" (UniqueName: \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.965668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965655 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.965865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.965865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965743 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:27.965865 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965759 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:27.966013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.965879 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\") pod \"e1f93a83-d39a-4070-8c39-94e94afe1d31\" (UID: \"e1f93a83-d39a-4070-8c39-94e94afe1d31\") " Oct 30 06:16:27.966402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.966372 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:27.966716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.966684 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:27.966716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.966708 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:27.966716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.966717 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:27.969109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.969037 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:27.979563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.979529 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "kube-api-access-wwhl5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:27.979703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:27.979573 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs" pod "e1f93a83-d39a-4070-8c39-94e94afe1d31" (UID: "e1f93a83-d39a-4070-8c39-94e94afe1d31"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:28.070038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.069895 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1f93a83-d39a-4070-8c39-94e94afe1d31-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.070038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.069931 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwhl5\" (UniqueName: \"kubernetes.io/projected/e1f93a83-d39a-4070-8c39-94e94afe1d31-kube-api-access-wwhl5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.070038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.069946 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1f93a83-d39a-4070-8c39-94e94afe1d31-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.205832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.205792 6171 generic.go:356] "Generic (PLEG): container finished" podID="e1f93a83-d39a-4070-8c39-94e94afe1d31" containerID="c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02" exitCode=0 Oct 30 06:16:28.206025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.205859 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" Oct 30 06:16:28.206025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.205884 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" event={"ID":"e1f93a83-d39a-4070-8c39-94e94afe1d31","Type":"ContainerDied","Data":"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02"} Oct 30 06:16:28.206025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.205939 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds" event={"ID":"e1f93a83-d39a-4070-8c39-94e94afe1d31","Type":"ContainerDied","Data":"39c3b619f8676c2a5ccc3e97e3a2c6d56b7a8756deffa1df158c0ec3fc5aee96"} Oct 30 06:16:28.206025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.205962 6171 scope.go:122] "RemoveContainer" containerID="c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02" Oct 30 06:16:28.220274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.220255 6171 scope.go:122] "RemoveContainer" containerID="c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02" Oct 30 06:16:28.220621 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:28.220598 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02\": container with ID starting with c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02 not found: ID does not exist" containerID="c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02" Oct 30 06:16:28.220715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.220634 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02"} err="failed to get container status \"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02\": rpc error: code = NotFound desc = could not find container \"c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02\": container with ID starting with c215a0ca304eef91f607090fd5f50a6b0f58d341916ceca08198ac1980a35f02 not found: ID does not exist" Oct 30 06:16:28.277312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.277236 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:16:28.286341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.286301 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w5-d1bfce/auth-test-model-d1bfce-kserve-565b55d4d5-8z5ds"] Oct 30 06:16:28.305028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.304959 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" podUID="727f5d41-57ee-4842-9a24-80385472d761" containerName="main" containerID="cri-o://e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6" gracePeriod=45 Oct 30 06:16:28.626888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.626861 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:16:28.684055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.683960 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684027 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684192 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\" (UniqueName: \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684234 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684245 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:28.684472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684302 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684499 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\" (UniqueName: \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\") pod \"727f5d41-57ee-4842-9a24-80385472d761\" (UID: \"727f5d41-57ee-4842-9a24-80385472d761\") " Oct 30 06:16:28.684603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684523 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:28.684921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.684893 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:28.685240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.685214 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.685240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.685242 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.685434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.685256 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.687111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.686808 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:28.687479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.687453 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "kube-api-access-vclx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:28.687842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.687814 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs" pod "727f5d41-57ee-4842-9a24-80385472d761" (UID: "727f5d41-57ee-4842-9a24-80385472d761"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:28.739681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.739649 6171 scope.go:122] "RemoveContainer" containerID="5939fde15d43bb91f502c0ff16eef8d66806fc43f0deba0ac1fbfb38a04769b1" Oct 30 06:16:28.754177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.754154 6171 scope.go:122] "RemoveContainer" containerID="2ee0ee2bd346637a2d130aadd6e997145223f3267c5f54eb365a6701647237eb" Oct 30 06:16:28.766880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.766858 6171 scope.go:122] "RemoveContainer" containerID="bd6ab2d58b9e2d5bd7f9af05a8f7128de7b0a25948b31048287d77f81962b712" Oct 30 06:16:28.778825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.778795 6171 scope.go:122] "RemoveContainer" containerID="7810f340ec95a750bb979e5e7aa3cdf2e29de830aa2ebe706996f2a8b91cccb2" Oct 30 06:16:28.786913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.786886 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vclx2\" (UniqueName: \"kubernetes.io/projected/727f5d41-57ee-4842-9a24-80385472d761-kube-api-access-vclx2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.787014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.786914 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/727f5d41-57ee-4842-9a24-80385472d761-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:28.787014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:28.786929 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/727f5d41-57ee-4842-9a24-80385472d761-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:29.218725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.218693 6171 generic.go:356] "Generic (PLEG): container finished" podID="727f5d41-57ee-4842-9a24-80385472d761" containerID="e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6" exitCode=0 Oct 30 06:16:29.218927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.218750 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" Oct 30 06:16:29.218927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.218774 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" event={"ID":"727f5d41-57ee-4842-9a24-80385472d761","Type":"ContainerDied","Data":"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6"} Oct 30 06:16:29.218927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.218832 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt" event={"ID":"727f5d41-57ee-4842-9a24-80385472d761","Type":"ContainerDied","Data":"a60e4625d7991ddb525666266cbeffc79ef9e5171925e72a544f9aff5d535b74"} Oct 30 06:16:29.218927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.218858 6171 scope.go:122] "RemoveContainer" containerID="e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6" Oct 30 06:16:29.232784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.232764 6171 scope.go:122] "RemoveContainer" containerID="e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6" Oct 30 06:16:29.233091 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:29.233064 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6\": container with ID starting with e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6 not found: ID does not exist" containerID="e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6" Oct 30 06:16:29.233176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.233100 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6"} err="failed to get container status \"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6\": rpc error: code = NotFound desc = could not find container \"e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6\": container with ID starting with e5ad5dfc0d01994aa54fec7595749e2fd62267474b93352a399d02c7acc69de6 not found: ID does not exist" Oct 30 06:16:29.280352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.280307 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:16:29.298020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.297989 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w5-91f42c/auth-test-model-91f42c-kserve-7b57b786d5-rvdgt"] Oct 30 06:16:29.639544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.639502 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" podUID="cff9f80c-25cf-494a-a0db-4d5290197e30" containerName="payload-processing" containerID="cri-o://0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a" gracePeriod=25 Oct 30 06:16:29.680982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.680923 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727f5d41-57ee-4842-9a24-80385472d761" path="/var/lib/kubelet/pods/727f5d41-57ee-4842-9a24-80385472d761/volumes" Oct 30 06:16:29.681698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:29.681673 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1f93a83-d39a-4070-8c39-94e94afe1d31" path="/var/lib/kubelet/pods/e1f93a83-d39a-4070-8c39-94e94afe1d31/volumes" Oct 30 06:16:30.009181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.009142 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:16:30.132420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.132375 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\") pod \"cff9f80c-25cf-494a-a0db-4d5290197e30\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " Oct 30 06:16:30.132633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.132610 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\" (UniqueName: \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\") pod \"cff9f80c-25cf-494a-a0db-4d5290197e30\" (UID: \"cff9f80c-25cf-494a-a0db-4d5290197e30\") " Oct 30 06:16:30.132833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.132809 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume" pod "cff9f80c-25cf-494a-a0db-4d5290197e30" (UID: "cff9f80c-25cf-494a-a0db-4d5290197e30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:30.133269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.133249 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cff9f80c-25cf-494a-a0db-4d5290197e30-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:30.135011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.134980 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8" pod "cff9f80c-25cf-494a-a0db-4d5290197e30" (UID: "cff9f80c-25cf-494a-a0db-4d5290197e30"). InnerVolumeSpecName "kube-api-access-pqwj8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:30.233598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.233560 6171 generic.go:356] "Generic (PLEG): container finished" podID="cff9f80c-25cf-494a-a0db-4d5290197e30" containerID="0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a" exitCode=0 Oct 30 06:16:30.233920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.233626 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" Oct 30 06:16:30.234116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.233651 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" event={"ID":"cff9f80c-25cf-494a-a0db-4d5290197e30","Type":"ContainerDied","Data":"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a"} Oct 30 06:16:30.234207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.234104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7" event={"ID":"cff9f80c-25cf-494a-a0db-4d5290197e30","Type":"ContainerDied","Data":"5ca43ef0a110c2b179533684efe4bed569bd4fb54b68bd901ddea876cfb4c65d"} Oct 30 06:16:30.234207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.234154 6171 scope.go:122] "RemoveContainer" containerID="0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a" Oct 30 06:16:30.234610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.234584 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqwj8\" (UniqueName: \"kubernetes.io/projected/cff9f80c-25cf-494a-a0db-4d5290197e30-kube-api-access-pqwj8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:30.246934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.246908 6171 scope.go:122] "RemoveContainer" containerID="0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a" Oct 30 06:16:30.247354 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:30.247317 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a\": container with ID starting with 0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a not found: ID does not exist" containerID="0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a" Oct 30 06:16:30.247477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.247360 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a"} err="failed to get container status \"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a\": rpc error: code = NotFound desc = could not find container \"0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a\": container with ID starting with 0e93143f5690c9eb91357ad5f82b1c585bdde3ccb90091ee99faadc9a9963c6a not found: ID does not exist" Oct 30 06:16:30.296244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.296211 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:16:30.302241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:30.302211 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-cc959cd95-xckd7"] Oct 30 06:16:31.245363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.245334 6171 generic.go:356] "Generic (PLEG): container finished" podID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerID="2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686" exitCode=0 Oct 30 06:16:31.245845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.245409 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerDied","Data":"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686"} Oct 30 06:16:31.245937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.245920 6171 scope.go:122] "RemoveContainer" containerID="2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686" Oct 30 06:16:31.247362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.247335 6171 generic.go:356] "Generic (PLEG): container finished" podID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerID="bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f" exitCode=0 Oct 30 06:16:31.247489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.247398 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerDied","Data":"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f"} Oct 30 06:16:31.247804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.247785 6171 scope.go:122] "RemoveContainer" containerID="bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f" Oct 30 06:16:31.248191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.248159 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.248323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.248239 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.248323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.248279 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.250019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.249989 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.250114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.250088 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.250170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.250132 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:16:31.417397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.417366 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:16:31.695684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.695575 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cff9f80c-25cf-494a-a0db-4d5290197e30" path="/var/lib/kubelet/pods/cff9f80c-25cf-494a-a0db-4d5290197e30/volumes" Oct 30 06:16:31.754792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.754752 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8"] Oct 30 06:16:31.755696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755672 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e1f93a83-d39a-4070-8c39-94e94afe1d31" containerName="main" Oct 30 06:16:31.755696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755691 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1f93a83-d39a-4070-8c39-94e94afe1d31" containerName="main" Oct 30 06:16:31.755696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755704 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cff9f80c-25cf-494a-a0db-4d5290197e30" containerName="payload-processing" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755709 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cff9f80c-25cf-494a-a0db-4d5290197e30" containerName="payload-processing" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755717 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9f59fe6b-a375-46a5-bc76-8185caf669be" containerName="revoke-keys" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755723 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f59fe6b-a375-46a5-bc76-8185caf669be" containerName="revoke-keys" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755749 6171 cpu_manager.go:414] "containerMap: removing container" podUID="727f5d41-57ee-4842-9a24-80385472d761" containerName="main" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755754 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727f5d41-57ee-4842-9a24-80385472d761" containerName="main" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755879 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="727f5d41-57ee-4842-9a24-80385472d761" containerName="main" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755897 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1f93a83-d39a-4070-8c39-94e94afe1d31" containerName="main" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755904 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cff9f80c-25cf-494a-a0db-4d5290197e30" containerName="payload-processing" Oct 30 06:16:31.755995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.755910 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f59fe6b-a375-46a5-bc76-8185caf669be" containerName="revoke-keys" Oct 30 06:16:31.761147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.761125 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.779132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.779107 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:31.779267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.779120 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:16:31.787080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.786297 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8"] Oct 30 06:16:31.878883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.878823 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bdnn\" (UniqueName: \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.879332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.879304 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.879661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.879635 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.981298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.981265 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.981490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.981354 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bdnn\" (UniqueName: \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.981560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.981498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.982162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.982131 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:31.984166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:31.984144 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:32.009926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.009899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bdnn\" (UniqueName: \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\") pod \"maas-api-revoke-sub-89c98f1e-rr5k8\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:32.082550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.082520 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:32.269129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.269018 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerStarted","Data":"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575"} Oct 30 06:16:32.271816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.271785 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerStarted","Data":"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2"} Oct 30 06:16:32.440555 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:32.440325 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca52e85b_1741_4e9d_900a_32c3c5f8c4ed.slice/crio-8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5 WatchSource:0}: Error finding container 8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5: Status 404 returned error can't find the container with id 8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5 Oct 30 06:16:32.449614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.449586 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8"] Oct 30 06:16:32.707119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.697468 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:16:32.707119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.698996 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.707119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.699064 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:32.879128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.879091 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:16:32.879879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.879853 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.880002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:32.879910 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:33.393153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.391855 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" event={"ID":"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed","Type":"ContainerStarted","Data":"68f83adf23840e5aa952199dbd8adad15635e3635e58ed09458223a4c3e239a6"} Oct 30 06:16:33.393153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.391926 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" event={"ID":"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed","Type":"ContainerStarted","Data":"8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5"} Oct 30 06:16:33.479001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.478934 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" podStartSLOduration=2.478914994 podStartE2EDuration="2.478914994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:33.476518898 +0000 UTC m=+2651.916896083" watchObservedRunningTime="2026-10-30 06:16:33.478914994 +0000 UTC m=+2651.919292187" Oct 30 06:16:33.709942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.705217 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.709942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.705276 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:33.879194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.879148 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.879398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:33.879238 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:34.448756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.448719 6171 generic.go:356] "Generic (PLEG): container finished" podID="ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" containerID="68f83adf23840e5aa952199dbd8adad15635e3635e58ed09458223a4c3e239a6" exitCode=0 Oct 30 06:16:34.449307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.448803 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" event={"ID":"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed","Type":"ContainerDied","Data":"68f83adf23840e5aa952199dbd8adad15635e3635e58ed09458223a4c3e239a6"} Oct 30 06:16:34.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.703839 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.711129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.703916 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:34.879369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.879333 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.879549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:34.879398 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:35.710920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.710831 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.713223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.711506 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:35.845651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.845623 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:35.879999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.879973 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.880237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.880025 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:35.905070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.904240 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\") pod \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " Oct 30 06:16:35.905070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.904537 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\" (UniqueName: \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\") pod \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " Oct 30 06:16:35.905070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.904715 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\") pod \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\" (UID: \"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed\") " Oct 30 06:16:35.909117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.905422 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca" pod "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" (UID: "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:35.925074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.923886 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn" pod "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" (UID: "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed"). InnerVolumeSpecName "kube-api-access-8bdnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:35.925074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.923988 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca" pod "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" (UID: "ca52e85b-1741-4e9d-900a-32c3c5f8c4ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:35.953182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.944671 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:35.953182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.944715 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bdnn\" (UniqueName: \"kubernetes.io/projected/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-kube-api-access-8bdnn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:35.953182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:35.944731 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:36.342413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.342379 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:16:36.375163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.375130 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:16:36.547121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.538111 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" event={"ID":"ca52e85b-1741-4e9d-900a-32c3c5f8c4ed","Type":"ContainerDied","Data":"8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5"} Oct 30 06:16:36.547121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.538178 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8bba9b9c8839f1648d780bd4c0a80c54925ddd6f8ec13bed26b5a577a0c567a5" Oct 30 06:16:36.547121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.538247 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8" Oct 30 06:16:36.736246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.732984 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.736246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.733056 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:36.892524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.892485 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.892869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:36.892834 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:37.551518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:37.551487 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:16:37.699393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:37.699357 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.699673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:37.699643 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.84:15021/healthz/ready\": dial tcp 10.133.1.84:15021: connect: connection refused" Oct 30 06:16:37.888131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:37.879845 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.888131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:37.879916 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.85:15021/healthz/ready\": dial tcp 10.133.1.85:15021: connect: connection refused" Oct 30 06:16:38.703667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:38.703635 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:16:38.704194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:38.704176 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:16:38.704834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:38.704812 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:16:38.886485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:38.886451 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:16:38.887613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:38.887592 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:16:41.343400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.343363 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" podUID="8164b629-629e-4c82-885e-ce6b6a0c29f9" containerName="payload-processing" containerID="cri-o://5fa3bdb9b4be7b2447a0abfbdc986d0e3839c35842fe215792c8c276de161510" gracePeriod=25 Oct 30 06:16:41.375481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.375445 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" podUID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" containerName="payload-pre-processing" containerID="cri-o://65a82a65c79916992a4b412acf34c1dbe1d979b8751ebfc88e6c47d36fc2b9b7" gracePeriod=25 Oct 30 06:16:41.603768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.603675 6171 generic.go:356] "Generic (PLEG): container finished" podID="8164b629-629e-4c82-885e-ce6b6a0c29f9" containerID="5fa3bdb9b4be7b2447a0abfbdc986d0e3839c35842fe215792c8c276de161510" exitCode=0 Oct 30 06:16:41.603937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.603817 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" event={"ID":"8164b629-629e-4c82-885e-ce6b6a0c29f9","Type":"ContainerDied","Data":"5fa3bdb9b4be7b2447a0abfbdc986d0e3839c35842fe215792c8c276de161510"} Oct 30 06:16:41.607150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.607119 6171 generic.go:356] "Generic (PLEG): container finished" podID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" containerID="65a82a65c79916992a4b412acf34c1dbe1d979b8751ebfc88e6c47d36fc2b9b7" exitCode=0 Oct 30 06:16:41.607436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.607411 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" event={"ID":"99d0948a-d43f-4bee-9f6b-d06d990a7d2f","Type":"ContainerDied","Data":"65a82a65c79916992a4b412acf34c1dbe1d979b8751ebfc88e6c47d36fc2b9b7"} Oct 30 06:16:41.688680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.688650 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:16:41.725513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.725491 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:16:41.803881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.803851 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\") pod \"8164b629-629e-4c82-885e-ce6b6a0c29f9\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " Oct 30 06:16:41.804088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.803895 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\" (UniqueName: \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\") pod \"8164b629-629e-4c82-885e-ce6b6a0c29f9\" (UID: \"8164b629-629e-4c82-885e-ce6b6a0c29f9\") " Oct 30 06:16:41.804397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.804370 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume" pod "8164b629-629e-4c82-885e-ce6b6a0c29f9" (UID: "8164b629-629e-4c82-885e-ce6b6a0c29f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:41.805205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.805174 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8164b629-629e-4c82-885e-ce6b6a0c29f9-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:41.806702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.806677 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm" pod "8164b629-629e-4c82-885e-ce6b6a0c29f9" (UID: "8164b629-629e-4c82-885e-ce6b6a0c29f9"). InnerVolumeSpecName "kube-api-access-pl8xm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:41.906032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.905952 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\" (UniqueName: \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\") pod \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " Oct 30 06:16:41.906032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.905993 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\") pod \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\" (UID: \"99d0948a-d43f-4bee-9f6b-d06d990a7d2f\") " Oct 30 06:16:41.906371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.906347 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume" pod "99d0948a-d43f-4bee-9f6b-d06d990a7d2f" (UID: "99d0948a-d43f-4bee-9f6b-d06d990a7d2f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:41.906440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.906396 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl8xm\" (UniqueName: \"kubernetes.io/projected/8164b629-629e-4c82-885e-ce6b6a0c29f9-kube-api-access-pl8xm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:41.908421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:41.908400 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm" pod "99d0948a-d43f-4bee-9f6b-d06d990a7d2f" (UID: "99d0948a-d43f-4bee-9f6b-d06d990a7d2f"). InnerVolumeSpecName "kube-api-access-828fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:42.007631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.007592 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-828fm\" (UniqueName: \"kubernetes.io/projected/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-kube-api-access-828fm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:42.007631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.007624 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0948a-d43f-4bee-9f6b-d06d990a7d2f-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:16:42.619749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.619660 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" Oct 30 06:16:42.619749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.619658 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2" event={"ID":"8164b629-629e-4c82-885e-ce6b6a0c29f9","Type":"ContainerDied","Data":"2a5f5430b056ed23b3f3cfd3a8e183e49009bb8885eac871aa22efae0c7ea6b9"} Oct 30 06:16:42.620312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.619829 6171 scope.go:122] "RemoveContainer" containerID="5fa3bdb9b4be7b2447a0abfbdc986d0e3839c35842fe215792c8c276de161510" Oct 30 06:16:42.622367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.622331 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" Oct 30 06:16:42.622367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.622359 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl" event={"ID":"99d0948a-d43f-4bee-9f6b-d06d990a7d2f","Type":"ContainerDied","Data":"24e763b111b486bc5185e101bd0d89505f111dcbea174e86f0e672967be3b527"} Oct 30 06:16:42.634944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.634926 6171 scope.go:122] "RemoveContainer" containerID="65a82a65c79916992a4b412acf34c1dbe1d979b8751ebfc88e6c47d36fc2b9b7" Oct 30 06:16:42.688303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.688282 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:16:42.721406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.721382 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-68fc5cdb99-pckhl"] Oct 30 06:16:42.790621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.790598 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:16:42.801704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:42.801680 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-79948bcdf7-2q6n2"] Oct 30 06:16:43.675825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:43.675790 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8164b629-629e-4c82-885e-ce6b6a0c29f9" path="/var/lib/kubelet/pods/8164b629-629e-4c82-885e-ce6b6a0c29f9/volumes" Oct 30 06:16:43.676347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:43.676317 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" path="/var/lib/kubelet/pods/99d0948a-d43f-4bee-9f6b-d06d990a7d2f/volumes" Oct 30 06:16:45.049378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:45.049340 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4"] Oct 30 06:16:45.061444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:45.061414 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4431ec71-rs2h4"] Oct 30 06:16:45.675145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:45.675110 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08d91c81-6a94-48ed-9760-8d57f51870e9" path="/var/lib/kubelet/pods/08d91c81-6a94-48ed-9760-8d57f51870e9/volumes" Oct 30 06:16:46.504563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:46.504532 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:16:51.184200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.184119 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:16:51.409919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.409891 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8164b629-629e-4c82-885e-ce6b6a0c29f9" containerName="payload-processing" Oct 30 06:16:51.410147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410100 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8164b629-629e-4c82-885e-ce6b6a0c29f9" containerName="payload-processing" Oct 30 06:16:51.410237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410155 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" containerName="revoke-keys" Oct 30 06:16:51.410237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410163 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" containerName="revoke-keys" Oct 30 06:16:51.410237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410218 6171 cpu_manager.go:414] "containerMap: removing container" podUID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" containerName="payload-pre-processing" Oct 30 06:16:51.410237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410227 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" containerName="payload-pre-processing" Oct 30 06:16:51.410512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410488 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" containerName="revoke-keys" Oct 30 06:16:51.410646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410525 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d0948a-d43f-4bee-9f6b-d06d990a7d2f" containerName="payload-pre-processing" Oct 30 06:16:51.410646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.410554 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8164b629-629e-4c82-885e-ce6b6a0c29f9" containerName="payload-processing" Oct 30 06:16:51.416491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.416469 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.419997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.419965 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:16:51.420183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.420165 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:16:51.433312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.433282 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:16:51.433472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.433343 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.437153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.437095 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2d4576\"" Oct 30 06:16:51.437153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.437096 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2d4576-dockercfg-8544v\"" Oct 30 06:16:51.466679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.466655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.466804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.466688 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5thkm\" (UniqueName: \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.466804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.466707 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj7rl\" (UniqueName: \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.466933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.466862 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.568522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.568494 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.568681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.568530 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5thkm\" (UniqueName: \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.568681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.568551 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj7rl\" (UniqueName: \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.568681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.568597 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.569243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.569221 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.569393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.569373 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.594339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.594306 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5thkm\" (UniqueName: \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\") pod \"payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.594507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.594483 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj7rl\" (UniqueName: \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\") pod \"payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.738594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.738569 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:51.750575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.750547 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:51.922966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.922923 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:16:51.932776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.932743 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.945716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.945648 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:16:51.953447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.952367 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-25d881\"/\"default-dockercfg-8xzbj\"" Oct 30 06:16:51.953447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.952414 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-25d881\"/\"openshift-service-ca.crt\"" Oct 30 06:16:51.953447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.952477 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-25d881\"/\"kube-root-ca.crt\"" Oct 30 06:16:51.953447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.952551 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-25d881\"/\"auth-test-model-25d881-kserve-self-signed-certs\"" Oct 30 06:16:51.975516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975487 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.975687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975555 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.975687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975676 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.975822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsv4g\" (UniqueName: \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.975884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:51.975955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:51.975891 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.068270 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:52.067704 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod852fdab1_5b35_4059_bfcd_be832e68ad4f.slice/crio-e3abf719505ca70e12fe50554d193a9f796de1d65c612ce6b257e09681e215f1 WatchSource:0}: Error finding container e3abf719505ca70e12fe50554d193a9f796de1d65c612ce6b257e09681e215f1: Status 404 returned error can't find the container with id e3abf719505ca70e12fe50554d193a9f796de1d65c612ce6b257e09681e215f1 Oct 30 06:16:52.080615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.080584 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.080860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.080840 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.081013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.080998 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.081211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.081185 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.081334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.081303 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.081409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.081356 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsv4g\" (UniqueName: \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.083737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.082309 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.083737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.082579 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.083737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.082829 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.088427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.088403 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:16:52.089791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.089769 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.090207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.089975 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.098091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.098064 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:16:52.109740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.109719 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsv4g\" (UniqueName: \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\") pod \"auth-test-model-25d881-kserve-77cb5b78fc-5dxmp\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.257155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.256844 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:52.725067 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:16:52.725020 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc0ab1062_bade_46bb_906a_6e92a09d12f1.slice/crio-38b594f2ecd7fbc80c71a2e206a6409206bc41cb34b0a0b7cf76451ef5a5b218 WatchSource:0}: Error finding container 38b594f2ecd7fbc80c71a2e206a6409206bc41cb34b0a0b7cf76451ef5a5b218: Status 404 returned error can't find the container with id 38b594f2ecd7fbc80c71a2e206a6409206bc41cb34b0a0b7cf76451ef5a5b218 Oct 30 06:16:52.725440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.725415 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:16:52.767563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.767532 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" event={"ID":"4dd37103-8329-4ecc-a977-a85065cc94ed","Type":"ContainerStarted","Data":"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901"} Oct 30 06:16:52.767690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.767590 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" event={"ID":"4dd37103-8329-4ecc-a977-a85065cc94ed","Type":"ContainerStarted","Data":"e940262ef1e39c12b39a54422833954c97e9ab09c786cb53ca95e0cd63b80eb4"} Oct 30 06:16:52.767690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.767636 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:16:52.769556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.769533 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" event={"ID":"852fdab1-5b35-4059-bfcd-be832e68ad4f","Type":"ContainerStarted","Data":"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada"} Oct 30 06:16:52.769677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.769572 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" event={"ID":"852fdab1-5b35-4059-bfcd-be832e68ad4f","Type":"ContainerStarted","Data":"e3abf719505ca70e12fe50554d193a9f796de1d65c612ce6b257e09681e215f1"} Oct 30 06:16:52.769677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.769615 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:16:52.770985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.770962 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" event={"ID":"c0ab1062-bade-46bb-906a-6e92a09d12f1","Type":"ContainerStarted","Data":"38b594f2ecd7fbc80c71a2e206a6409206bc41cb34b0a0b7cf76451ef5a5b218"} Oct 30 06:16:52.818562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.818509 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" podStartSLOduration=1.818492412 podStartE2EDuration="1.818492412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:52.812740536 +0000 UTC m=+2671.253117728" watchObservedRunningTime="2026-10-30 06:16:52.818492412 +0000 UTC m=+2671.258869607" Oct 30 06:16:52.862349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:52.862291 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" podStartSLOduration=1.862273993 podStartE2EDuration="1.862273993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:52.859407254 +0000 UTC m=+2671.299784448" watchObservedRunningTime="2026-10-30 06:16:52.862273993 +0000 UTC m=+2671.302651189" Oct 30 06:16:53.782567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:53.782520 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" event={"ID":"c0ab1062-bade-46bb-906a-6e92a09d12f1","Type":"ContainerStarted","Data":"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b"} Oct 30 06:16:53.840311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:53.840261 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" podStartSLOduration=2.840248526 podStartE2EDuration="2.840248526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:53.836825312 +0000 UTC m=+2672.277202518" watchObservedRunningTime="2026-10-30 06:16:53.840248526 +0000 UTC m=+2672.280625748" Oct 30 06:16:54.257468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:54.257437 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:16:55.038758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:55.038728 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9"] Oct 30 06:16:55.052575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:55.052547 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5efbc29c-vxsl9"] Oct 30 06:16:55.688412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:16:55.688377 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b101ad-1b9d-4616-9203-22ca69c49b3d" path="/var/lib/kubelet/pods/92b101ad-1b9d-4616-9203-22ca69c49b3d/volumes" Oct 30 06:16:56.382753 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:16:56.382717 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:17:01.527803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.527759 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" podUID="4fe47436-0fbf-403d-8b56-9485a5572b27" containerName="main" containerID="cri-o://7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229" gracePeriod=45 Oct 30 06:17:01.819008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.818983 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:17:01.861845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.861804 6171 generic.go:356] "Generic (PLEG): container finished" podID="4fe47436-0fbf-403d-8b56-9485a5572b27" containerID="7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229" exitCode=0 Oct 30 06:17:01.862020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.861866 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" Oct 30 06:17:01.862020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.861887 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" event={"ID":"4fe47436-0fbf-403d-8b56-9485a5572b27","Type":"ContainerDied","Data":"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229"} Oct 30 06:17:01.862020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.861930 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq" event={"ID":"4fe47436-0fbf-403d-8b56-9485a5572b27","Type":"ContainerDied","Data":"59d61d438df05aaf0cf2c9a0ca66451f0415cfc3698551ea49bced8ece3decf9"} Oct 30 06:17:01.862020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.861941 6171 scope.go:122] "RemoveContainer" containerID="7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229" Oct 30 06:17:01.876992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.876968 6171 scope.go:122] "RemoveContainer" containerID="7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229" Oct 30 06:17:01.877454 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:17:01.877422 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229\": container with ID starting with 7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229 not found: ID does not exist" containerID="7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229" Oct 30 06:17:01.877550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.877464 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229"} err="failed to get container status \"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229\": rpc error: code = NotFound desc = could not find container \"7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229\": container with ID starting with 7cdf90b5335a73e9660cbd57572e85b4395bffa05330f9fdc86357867cd3e229 not found: ID does not exist" Oct 30 06:17:01.961667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.961634 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.961855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.961737 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.961855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.961758 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\" (UniqueName: \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.961997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.961944 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.961997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.961982 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.962176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.962087 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:01.962176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.962139 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\") pod \"4fe47436-0fbf-403d-8b56-9485a5572b27\" (UID: \"4fe47436-0fbf-403d-8b56-9485a5572b27\") " Oct 30 06:17:01.962301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.962279 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:01.962424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.962388 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:01.963282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.963257 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:01.963374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.963287 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:01.963374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.963300 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:01.964170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.964141 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:01.964310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.964290 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:01.964401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:01.964379 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c" pod "4fe47436-0fbf-403d-8b56-9485a5572b27" (UID: "4fe47436-0fbf-403d-8b56-9485a5572b27"). InnerVolumeSpecName "kube-api-access-sbm9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:02.064835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.064747 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4fe47436-0fbf-403d-8b56-9485a5572b27-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:02.064835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.064782 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbm9c\" (UniqueName: \"kubernetes.io/projected/4fe47436-0fbf-403d-8b56-9485a5572b27-kube-api-access-sbm9c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:02.064835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.064794 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4fe47436-0fbf-403d-8b56-9485a5572b27-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:02.216187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.216145 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:17:02.229466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.229432 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-24db22/auth-test-model-24db22-kserve-6bb4d9d85c-p9szq"] Oct 30 06:17:02.258313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.258275 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:17:02.272390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.272363 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:17:02.886654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:02.886621 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:17:03.675769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:03.675733 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fe47436-0fbf-403d-8b56-9485a5572b27" path="/var/lib/kubelet/pods/4fe47436-0fbf-403d-8b56-9485a5572b27/volumes" Oct 30 06:17:03.784697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:03.784668 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:17:03.784883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:03.784763 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:17:06.035645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:06.035608 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q"] Oct 30 06:17:06.048528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:06.048504 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79b306a9-sgx9q"] Oct 30 06:17:07.316099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.316069 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj"] Oct 30 06:17:07.316961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.316945 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4fe47436-0fbf-403d-8b56-9485a5572b27" containerName="main" Oct 30 06:17:07.317012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.316962 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fe47436-0fbf-403d-8b56-9485a5572b27" containerName="main" Oct 30 06:17:07.317148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.317137 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fe47436-0fbf-403d-8b56-9485a5572b27" containerName="main" Oct 30 06:17:07.323212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.323193 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.341607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.341581 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj"] Oct 30 06:17:07.347825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.347799 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:17:07.347968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.347952 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:17:07.398182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.398147 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.398404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.398209 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.398562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.398531 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwgkv\" (UniqueName: \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.499870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.499837 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwgkv\" (UniqueName: \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.500105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.500033 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.500197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.500116 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.500800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.500782 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.502673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.502647 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.525744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.525719 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwgkv\" (UniqueName: \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\") pod \"maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.639530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.639453 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:07.679098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.679033 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02" path="/var/lib/kubelet/pods/9e88e9bf-6f7c-4b7d-bd9e-df554a49ed02/volumes" Oct 30 06:17:07.965648 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:07.965620 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b648fd_f366_474c_9bcd_a16506447309.slice/crio-856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14 WatchSource:0}: Error finding container 856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14: Status 404 returned error can't find the container with id 856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14 Oct 30 06:17:07.967203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:07.967177 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj"] Oct 30 06:17:08.964298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:08.964264 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" event={"ID":"d1b648fd-f366-474c-9bcd-a16506447309","Type":"ContainerStarted","Data":"95eff2ef76e6b82df1ea1e104235402e61984c278625e892d84eb414ff9ec2e4"} Oct 30 06:17:08.964712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:08.964319 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" event={"ID":"d1b648fd-f366-474c-9bcd-a16506447309","Type":"ContainerStarted","Data":"856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14"} Oct 30 06:17:09.012359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:09.012287 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" podStartSLOduration=2.012268573 podStartE2EDuration="2.012268573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:09.008483247 +0000 UTC m=+2687.448860440" watchObservedRunningTime="2026-10-30 06:17:09.012268573 +0000 UTC m=+2687.452645767" Oct 30 06:17:09.690352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:09.690323 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:17:09.977563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:09.977527 6171 generic.go:356] "Generic (PLEG): container finished" podID="d1b648fd-f366-474c-9bcd-a16506447309" containerID="95eff2ef76e6b82df1ea1e104235402e61984c278625e892d84eb414ff9ec2e4" exitCode=0 Oct 30 06:17:09.978001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:09.977591 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" event={"ID":"d1b648fd-f366-474c-9bcd-a16506447309","Type":"ContainerDied","Data":"95eff2ef76e6b82df1ea1e104235402e61984c278625e892d84eb414ff9ec2e4"} Oct 30 06:17:11.183621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.183587 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:11.276121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.275259 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\" (UniqueName: \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\") pod \"d1b648fd-f366-474c-9bcd-a16506447309\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " Oct 30 06:17:11.276121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.275414 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\") pod \"d1b648fd-f366-474c-9bcd-a16506447309\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " Oct 30 06:17:11.276121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.275522 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\") pod \"d1b648fd-f366-474c-9bcd-a16506447309\" (UID: \"d1b648fd-f366-474c-9bcd-a16506447309\") " Oct 30 06:17:11.301070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.281612 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca" pod "d1b648fd-f366-474c-9bcd-a16506447309" (UID: "d1b648fd-f366-474c-9bcd-a16506447309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:11.301070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.284297 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:11.327278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.327217 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca" pod "d1b648fd-f366-474c-9bcd-a16506447309" (UID: "d1b648fd-f366-474c-9bcd-a16506447309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:11.350303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.350236 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv" pod "d1b648fd-f366-474c-9bcd-a16506447309" (UID: "d1b648fd-f366-474c-9bcd-a16506447309"). InnerVolumeSpecName "kube-api-access-zwgkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:11.401205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.401162 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwgkv\" (UniqueName: \"kubernetes.io/projected/d1b648fd-f366-474c-9bcd-a16506447309-kube-api-access-zwgkv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:11.401464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:11.401445 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b648fd-f366-474c-9bcd-a16506447309-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:17:12.232828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:12.221567 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" Oct 30 06:17:12.232828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:12.221642 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj" event={"ID":"d1b648fd-f366-474c-9bcd-a16506447309","Type":"ContainerDied","Data":"856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14"} Oct 30 06:17:12.232828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:12.221734 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="856a7b101c680e65165a695f048847800d9b051ea7f94c63013ba4ef584a7f14" Oct 30 06:17:18.037037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.037003 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp"] Oct 30 06:17:18.050292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.050259 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ea494249-qbvcp"] Oct 30 06:17:18.796793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.786100 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:17:18.796793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.787352 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d1b648fd-f366-474c-9bcd-a16506447309" containerName="revoke-keys" Oct 30 06:17:18.796793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.787375 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b648fd-f366-474c-9bcd-a16506447309" containerName="revoke-keys" Oct 30 06:17:18.796793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.787640 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b648fd-f366-474c-9bcd-a16506447309" containerName="revoke-keys" Oct 30 06:17:18.815067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.801983 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:18.832838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.832810 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-bf87e1\"" Oct 30 06:17:18.835262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.835226 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-bf87e1-dockercfg-dzwts\"" Oct 30 06:17:18.836133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.836110 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:17:18.913261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.896679 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6hmkr\" (UniqueName: \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:18.913261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.896863 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:18.992708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:18.992670 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:17:19.001428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.001397 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.008316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.007952 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:17:19.008573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.008545 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.008723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.008682 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6hmkr\" (UniqueName: \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:19.008808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.008726 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjw5\" (UniqueName: \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.008870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.008823 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:19.016187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.015961 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:19.036905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.036878 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6hmkr\" (UniqueName: \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\") pod \"payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:19.110942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.110845 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjw5\" (UniqueName: \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.111446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.111186 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.112206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.112180 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.137238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.137209 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjw5\" (UniqueName: \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\") pod \"payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.216670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.216628 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:19.339468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.339135 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:19.533380 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:19.533348 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e47b155_107e_4652_a357_52cea322229e.slice/crio-070282c2ad5f80df439bb8db701f0625360b6ea42b9c73e6a49bf930176268f6 WatchSource:0}: Error finding container 070282c2ad5f80df439bb8db701f0625360b6ea42b9c73e6a49bf930176268f6: Status 404 returned error can't find the container with id 070282c2ad5f80df439bb8db701f0625360b6ea42b9c73e6a49bf930176268f6 Oct 30 06:17:19.533751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.533730 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:17:19.735856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.735829 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afe624e2-2de8-4819-981d-cb71566d1d24" path="/var/lib/kubelet/pods/afe624e2-2de8-4819-981d-cb71566d1d24/volumes" Oct 30 06:17:19.736231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:19.736215 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:17:20.331438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.331397 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" event={"ID":"5e47b155-107e-4652-a357-52cea322229e","Type":"ContainerStarted","Data":"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b"} Oct 30 06:17:20.331903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.331463 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" event={"ID":"5e47b155-107e-4652-a357-52cea322229e","Type":"ContainerStarted","Data":"070282c2ad5f80df439bb8db701f0625360b6ea42b9c73e6a49bf930176268f6"} Oct 30 06:17:20.331903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.331493 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:20.333404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.333377 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" event={"ID":"9d014002-9bed-4379-aea2-89b267686ae5","Type":"ContainerStarted","Data":"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea"} Oct 30 06:17:20.333540 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.333422 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" event={"ID":"9d014002-9bed-4379-aea2-89b267686ae5","Type":"ContainerStarted","Data":"575f56ba757df7a876ae7e552e84e976126a22f3deb1362ec8a76df116fa72fd"} Oct 30 06:17:20.333540 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.333467 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:20.381894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.381846 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" podStartSLOduration=2.381833764 podStartE2EDuration="2.381833764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:20.378458214 +0000 UTC m=+2698.818835407" watchObservedRunningTime="2026-10-30 06:17:20.381833764 +0000 UTC m=+2698.822210971" Oct 30 06:17:20.428531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:20.428483 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" podStartSLOduration=2.428470721 podStartE2EDuration="2.428470721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:20.423265006 +0000 UTC m=+2698.863642199" watchObservedRunningTime="2026-10-30 06:17:20.428470721 +0000 UTC m=+2698.868847914" Oct 30 06:17:26.382332 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:17:26.382291 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:17:28.068521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.068492 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w"] Oct 30 06:17:28.082222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.082195 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-43dfbf61-mvd4w"] Oct 30 06:17:28.796069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.795097 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:17:28.853080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.851476 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:28.853080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.852473 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:17:28.906068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.897213 6171 scope.go:122] "RemoveContainer" containerID="6bcaed7e7fe414f59ff4f9c64b1f5df438d4a8c44d563b7a50a53c3e26a427ae" Oct 30 06:17:28.971166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.971137 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-dfc3d2-dockercfg-wd7p5\"" Oct 30 06:17:28.971574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.971555 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-dfc3d2\"" Oct 30 06:17:29.025117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.989186 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdhr6\" (UniqueName: \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.025117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:28.989318 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.025117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.009381 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:17:29.034723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.032494 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:17:29.034723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.032643 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.059014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.058940 6171 scope.go:122] "RemoveContainer" containerID="424915c527a18a3c037936ab6d257fe260a80a2322a9654587242da6070212de" Oct 30 06:17:29.073398 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.073376 6171 scope.go:122] "RemoveContainer" containerID="b1975ba5bec4ebd230e55c20fffa877573dc52c6c5f64807946bfb0015f90eb1" Oct 30 06:17:29.087552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.087530 6171 scope.go:122] "RemoveContainer" containerID="fe96ab29b43d0717b26700310e0aea029b43ed34dd5c216003b9c97be0c3e9c2" Oct 30 06:17:29.090474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.090446 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdhr6\" (UniqueName: \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.090588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.090550 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.090655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.090601 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.090782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.090767 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.091490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.091470 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.101568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.101549 6171 scope.go:122] "RemoveContainer" containerID="416691d959eee124d468bdfb62a4d7032162fc3d3c54d3dc3e7a3ee260b375be" Oct 30 06:17:29.116219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.116199 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdhr6\" (UniqueName: \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\") pod \"payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.192831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.192793 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.193013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.192992 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.194097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.194072 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.219406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.219376 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\") pod \"payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.349711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.349638 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:29.365587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.365563 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:29.698422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.698332 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0acc2e5b-aeb1-4d71-8989-71aa4a2600a5" path="/var/lib/kubelet/pods/0acc2e5b-aeb1-4d71-8989-71aa4a2600a5/volumes" Oct 30 06:17:29.712852 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:29.712823 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2944f6bc_3205_419c_962a_0bdf56e52cd1.slice/crio-9387a0733a5a449abefc597a33c13daab8c7ef815219d07e5d2f22642a31cbbc WatchSource:0}: Error finding container 9387a0733a5a449abefc597a33c13daab8c7ef815219d07e5d2f22642a31cbbc: Status 404 returned error can't find the container with id 9387a0733a5a449abefc597a33c13daab8c7ef815219d07e5d2f22642a31cbbc Oct 30 06:17:29.714090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.714019 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:17:29.751488 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:29.751457 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7e273a_0d9a_4c69_9b6f_e565e8e4f847.slice/crio-9c5238593ef1717cc11225fd1d9c51966ae41d279f198c656b832e193ee23af2 WatchSource:0}: Error finding container 9c5238593ef1717cc11225fd1d9c51966ae41d279f198c656b832e193ee23af2: Status 404 returned error can't find the container with id 9c5238593ef1717cc11225fd1d9c51966ae41d279f198c656b832e193ee23af2 Oct 30 06:17:29.751908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:29.751884 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:17:30.456406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.456354 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" event={"ID":"2944f6bc-3205-419c-962a-0bdf56e52cd1","Type":"ContainerStarted","Data":"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58"} Oct 30 06:17:30.456956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.456431 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:30.456956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.456456 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" event={"ID":"2944f6bc-3205-419c-962a-0bdf56e52cd1","Type":"ContainerStarted","Data":"9387a0733a5a449abefc597a33c13daab8c7ef815219d07e5d2f22642a31cbbc"} Oct 30 06:17:30.458439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.458401 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" event={"ID":"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847","Type":"ContainerStarted","Data":"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9"} Oct 30 06:17:30.458546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.458452 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:30.458546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.458477 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" event={"ID":"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847","Type":"ContainerStarted","Data":"9c5238593ef1717cc11225fd1d9c51966ae41d279f198c656b832e193ee23af2"} Oct 30 06:17:30.502125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.502038 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" podStartSLOduration=2.502022888 podStartE2EDuration="2.502022888s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:30.501130378 +0000 UTC m=+2708.941507574" watchObservedRunningTime="2026-10-30 06:17:30.502022888 +0000 UTC m=+2708.942400085" Oct 30 06:17:30.551221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:30.551181 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" podStartSLOduration=2.551168382 podStartE2EDuration="2.551168382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:30.546410805 +0000 UTC m=+2708.986787998" watchObservedRunningTime="2026-10-30 06:17:30.551168382 +0000 UTC m=+2708.991545959" Oct 30 06:17:31.345795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:31.345763 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:17:31.345958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:31.345856 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:17:41.475232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:41.475200 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:17:41.475673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:41.475278 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:17:51.975476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:51.975441 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:17:51.980794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:51.980772 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:51.998806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:51.998183 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5deb36\"" Oct 30 06:17:51.998806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:51.998226 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5deb36-dockercfg-hwf4v\"" Oct 30 06:17:52.146071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.145829 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.146071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.145891 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjpn8\" (UniqueName: \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.231122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.217099 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:17:52.247067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.242159 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:17:52.293070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.288620 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.293070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.288686 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjpn8\" (UniqueName: \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.293070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.290443 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.348775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.345136 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjpn8\" (UniqueName: \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.348775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.346654 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.357058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.357016 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.365326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.364710 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:17:52.492611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.492502 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gd48\" (UniqueName: \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.493103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.493069 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.594266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.594234 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gd48\" (UniqueName: \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.594458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.594385 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.595187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.595165 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.620958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.620935 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gd48\" (UniqueName: \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\") pod \"payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.680235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.680212 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:52.684077 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:52.684033 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb664e835_8fa6_47c9_9791_c3cf95303019.slice/crio-e1d5bc33672260e2d53b6bc7002eaad10a1a6b19f5eee16ed4b0c7258911c4b9 WatchSource:0}: Error finding container e1d5bc33672260e2d53b6bc7002eaad10a1a6b19f5eee16ed4b0c7258911c4b9: Status 404 returned error can't find the container with id e1d5bc33672260e2d53b6bc7002eaad10a1a6b19f5eee16ed4b0c7258911c4b9 Oct 30 06:17:52.687206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.687136 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:17:52.816916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.816888 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" event={"ID":"b664e835-8fa6-47c9-9791-c3cf95303019","Type":"ContainerStarted","Data":"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd"} Oct 30 06:17:52.817086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.816942 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" event={"ID":"b664e835-8fa6-47c9-9791-c3cf95303019","Type":"ContainerStarted","Data":"e1d5bc33672260e2d53b6bc7002eaad10a1a6b19f5eee16ed4b0c7258911c4b9"} Oct 30 06:17:52.817086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.816973 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:17:52.866834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.866777 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" podStartSLOduration=1.866764224 podStartE2EDuration="1.866764224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:52.86142918 +0000 UTC m=+2731.301806378" watchObservedRunningTime="2026-10-30 06:17:52.866764224 +0000 UTC m=+2731.307141417" Oct 30 06:17:52.993986 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:17:52.993919 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9272efda_02aa_4f36_bbc5_5079c6019f76.slice/crio-d6bc6ccef419d4779c4815d615444b753e6a0a0a41304a525588137db03044db WatchSource:0}: Error finding container d6bc6ccef419d4779c4815d615444b753e6a0a0a41304a525588137db03044db: Status 404 returned error can't find the container with id d6bc6ccef419d4779c4815d615444b753e6a0a0a41304a525588137db03044db Oct 30 06:17:52.994773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:52.994750 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:17:53.827965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:53.827899 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" event={"ID":"9272efda-02aa-4f36-bbc5-5079c6019f76","Type":"ContainerStarted","Data":"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede"} Oct 30 06:17:53.828226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:53.827988 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:17:53.828226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:53.828016 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" event={"ID":"9272efda-02aa-4f36-bbc5-5079c6019f76","Type":"ContainerStarted","Data":"d6bc6ccef419d4779c4815d615444b753e6a0a0a41304a525588137db03044db"} Oct 30 06:17:53.876188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:17:53.876133 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" podStartSLOduration=2.876115119 podStartE2EDuration="2.876115119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:53.874783705 +0000 UTC m=+2732.315160904" watchObservedRunningTime="2026-10-30 06:17:53.876115119 +0000 UTC m=+2732.316492313" Oct 30 06:17:56.383729 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:17:56.383688 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:18:03.832533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:03.832501 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:18:04.870277 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:04.870239 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:18:06.220017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:06.218701 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:18:06.220017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:06.218976 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" containerID="cri-o://f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715" gracePeriod=30 Oct 30 06:18:07.785714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:07.785675 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:18:07.823428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:07.823391 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:18:09.594893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:09.594859 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj"] Oct 30 06:18:09.600172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:09.600143 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-db0540-2cvtj"] Oct 30 06:18:09.677122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:09.677095 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b648fd-f366-474c-9bcd-a16506447309" path="/var/lib/kubelet/pods/d1b648fd-f366-474c-9bcd-a16506447309/volumes" Oct 30 06:18:11.584563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.584533 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:18:11.704591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704559 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.704862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704604 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.704862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704711 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.704862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704849 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704902 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704943 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.704980 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705019 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705136 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\") pod \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\" (UID: \"3ffea1d6-cda5-4edb-a278-268881b3b1bb\") " Oct 30 06:18:11.705381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705261 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:11.705381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705278 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:11.705596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705554 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:11.705753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705727 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:11.705826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.705754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:11.707152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707127 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.707287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707155 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.707287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707168 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.707287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707181 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.707287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707196 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.707941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.707912 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:11.708493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.708467 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:11.708641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.708485 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:11.710334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.710313 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2" pod "3ffea1d6-cda5-4edb-a278-268881b3b1bb" (UID: "3ffea1d6-cda5-4edb-a278-268881b3b1bb"). InnerVolumeSpecName "kube-api-access-bjss2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:11.808478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.808383 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.808478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.808416 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.808478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.808429 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjss2\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-kube-api-access-bjss2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:11.808478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:11.808442 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffea1d6-cda5-4edb-a278-268881b3b1bb-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:12.299743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.299709 6171 generic.go:356] "Generic (PLEG): container finished" podID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerID="f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715" exitCode=0 Oct 30 06:18:12.299942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.299766 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" Oct 30 06:18:12.299942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.299764 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" event={"ID":"3ffea1d6-cda5-4edb-a278-268881b3b1bb","Type":"ContainerDied","Data":"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715"} Oct 30 06:18:12.299942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.299838 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5" event={"ID":"3ffea1d6-cda5-4edb-a278-268881b3b1bb","Type":"ContainerDied","Data":"02f1f71e2e6fe6f91c66619591897fe1abc70ad364d40d300056d6a9406a8b98"} Oct 30 06:18:12.299942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.299863 6171 scope.go:122] "RemoveContainer" containerID="f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715" Oct 30 06:18:12.315285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.315263 6171 scope.go:122] "RemoveContainer" containerID="f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715" Oct 30 06:18:12.315600 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:12.315571 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715\": container with ID starting with f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715 not found: ID does not exist" containerID="f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715" Oct 30 06:18:12.315696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.315610 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715"} err="failed to get container status \"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715\": rpc error: code = NotFound desc = could not find container \"f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715\": container with ID starting with f43daebfdf361c7cb8e2f0c287c66fda6b904ef73cd649333870707601ee9715 not found: ID does not exist" Oct 30 06:18:12.376768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.376743 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:18:12.381843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.381823 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-03d349-openshift-default-cdb65584c-q7qn5"] Oct 30 06:18:12.786648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.786612 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" podUID="4095fa43-781a-47a2-b7db-b39205b0a9b3" containerName="payload-processing" containerID="cri-o://23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad" gracePeriod=25 Oct 30 06:18:12.824244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:12.824206 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" podUID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" containerName="payload-pre-processing" containerID="cri-o://d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58" gracePeriod=25 Oct 30 06:18:13.132799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.132766 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:18:13.231630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.231586 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\") pod \"4095fa43-781a-47a2-b7db-b39205b0a9b3\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " Oct 30 06:18:13.231892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.231872 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\" (UniqueName: \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\") pod \"4095fa43-781a-47a2-b7db-b39205b0a9b3\" (UID: \"4095fa43-781a-47a2-b7db-b39205b0a9b3\") " Oct 30 06:18:13.233626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.233591 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume" pod "4095fa43-781a-47a2-b7db-b39205b0a9b3" (UID: "4095fa43-781a-47a2-b7db-b39205b0a9b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:13.239840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.237209 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2" pod "4095fa43-781a-47a2-b7db-b39205b0a9b3" (UID: "4095fa43-781a-47a2-b7db-b39205b0a9b3"). InnerVolumeSpecName "kube-api-access-l2fg2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:13.239840 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.237866 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:18:13.316285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.316099 6171 generic.go:356] "Generic (PLEG): container finished" podID="4095fa43-781a-47a2-b7db-b39205b0a9b3" containerID="23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad" exitCode=0 Oct 30 06:18:13.316285 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.316204 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" Oct 30 06:18:13.316516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.316496 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" event={"ID":"4095fa43-781a-47a2-b7db-b39205b0a9b3","Type":"ContainerDied","Data":"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad"} Oct 30 06:18:13.316567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.316547 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6" event={"ID":"4095fa43-781a-47a2-b7db-b39205b0a9b3","Type":"ContainerDied","Data":"3ff76c5ad2f6366ff878357b2b903f094c6a287ec187a4f727f7637829e17b82"} Oct 30 06:18:13.316624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.316576 6171 scope.go:122] "RemoveContainer" containerID="23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad" Oct 30 06:18:13.318665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.318627 6171 generic.go:356] "Generic (PLEG): container finished" podID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" containerID="d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58" exitCode=0 Oct 30 06:18:13.318856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.318714 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" event={"ID":"58567eaf-73da-45e5-9c46-1a2079d8aa0f","Type":"ContainerDied","Data":"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58"} Oct 30 06:18:13.318856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.318755 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" event={"ID":"58567eaf-73da-45e5-9c46-1a2079d8aa0f","Type":"ContainerDied","Data":"3525698d79c375521b492f7b31f215331505cf77060bbdcdf182a5e21a76967c"} Oct 30 06:18:13.318856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.318824 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969" Oct 30 06:18:13.337819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.337787 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\" (UniqueName: \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\") pod \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " Oct 30 06:18:13.337991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.337970 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\") pod \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\" (UID: \"58567eaf-73da-45e5-9c46-1a2079d8aa0f\") " Oct 30 06:18:13.338949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.338924 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2fg2\" (UniqueName: \"kubernetes.io/projected/4095fa43-781a-47a2-b7db-b39205b0a9b3-kube-api-access-l2fg2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:13.339035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.338956 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4095fa43-781a-47a2-b7db-b39205b0a9b3-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:13.339373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.339346 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume" pod "58567eaf-73da-45e5-9c46-1a2079d8aa0f" (UID: "58567eaf-73da-45e5-9c46-1a2079d8aa0f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:13.339578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.339555 6171 scope.go:122] "RemoveContainer" containerID="23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad" Oct 30 06:18:13.340137 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:13.340109 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad\": container with ID starting with 23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad not found: ID does not exist" containerID="23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad" Oct 30 06:18:13.340251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.340146 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad"} err="failed to get container status \"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad\": rpc error: code = NotFound desc = could not find container \"23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad\": container with ID starting with 23b2a386a96ad0d5b199e4c19df83a0069bffd9cfbb22398a5d505ef75a61fad not found: ID does not exist" Oct 30 06:18:13.340251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.340168 6171 scope.go:122] "RemoveContainer" containerID="d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58" Oct 30 06:18:13.342270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.342235 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl" pod "58567eaf-73da-45e5-9c46-1a2079d8aa0f" (UID: "58567eaf-73da-45e5-9c46-1a2079d8aa0f"). InnerVolumeSpecName "kube-api-access-cb2nl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:13.355327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.355302 6171 scope.go:122] "RemoveContainer" containerID="d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58" Oct 30 06:18:13.355630 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:13.355603 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58\": container with ID starting with d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58 not found: ID does not exist" containerID="d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58" Oct 30 06:18:13.355716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.355633 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58"} err="failed to get container status \"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58\": rpc error: code = NotFound desc = could not find container \"d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58\": container with ID starting with d82c27b8f388e84be625959b775d658b66f1f5ee195dcf17ecf7ab897064ea58 not found: ID does not exist" Oct 30 06:18:13.403766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.403738 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:18:13.408908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.408883 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-6b6957bd65-hl4j6"] Oct 30 06:18:13.440541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.440513 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb2nl\" (UniqueName: \"kubernetes.io/projected/58567eaf-73da-45e5-9c46-1a2079d8aa0f-kube-api-access-cb2nl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:13.440541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.440541 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58567eaf-73da-45e5-9c46-1a2079d8aa0f-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:13.681470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.681434 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" path="/var/lib/kubelet/pods/3ffea1d6-cda5-4edb-a278-268881b3b1bb/volumes" Oct 30 06:18:13.682218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.682191 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4095fa43-781a-47a2-b7db-b39205b0a9b3" path="/var/lib/kubelet/pods/4095fa43-781a-47a2-b7db-b39205b0a9b3/volumes" Oct 30 06:18:13.682654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.682634 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:18:13.687165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:13.687122 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-5f6dbd77dc-ps969"] Oct 30 06:18:15.676660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:15.676617 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" path="/var/lib/kubelet/pods/58567eaf-73da-45e5-9c46-1a2079d8aa0f/volumes" Oct 30 06:18:17.478367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.478333 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns"] Oct 30 06:18:17.479495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479475 6171 cpu_manager.go:414] "containerMap: removing container" podUID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" containerName="payload-pre-processing" Oct 30 06:18:17.479495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479495 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" containerName="payload-pre-processing" Oct 30 06:18:17.479648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479523 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" Oct 30 06:18:17.479648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479529 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" Oct 30 06:18:17.479648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479571 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4095fa43-781a-47a2-b7db-b39205b0a9b3" containerName="payload-processing" Oct 30 06:18:17.479648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479575 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4095fa43-781a-47a2-b7db-b39205b0a9b3" containerName="payload-processing" Oct 30 06:18:17.479789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479735 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4095fa43-781a-47a2-b7db-b39205b0a9b3" containerName="payload-processing" Oct 30 06:18:17.479789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479751 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="58567eaf-73da-45e5-9c46-1a2079d8aa0f" containerName="payload-pre-processing" Oct 30 06:18:17.479789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.479760 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffea1d6-cda5-4edb-a278-268881b3b1bb" containerName="istio-proxy" Oct 30 06:18:17.485711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.485689 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.506071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.506031 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns"] Oct 30 06:18:17.592647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.592585 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:18:17.592862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.592666 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:18:17.716053 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.716018 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.716260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.716233 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcn27\" (UniqueName: \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.716462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.716444 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.817587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.817510 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcn27\" (UniqueName: \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.817738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.817700 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.817821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.817805 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.818420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.818395 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.820236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.820216 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:17.843193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:17.843161 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcn27\" (UniqueName: \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\") pod \"maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:18.101644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:18.101567 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:18.418212 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:18:18.418184 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c2a5b71_eec1_4798_8f68_94d4d4046e6e.slice/crio-8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf WatchSource:0}: Error finding container 8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf: Status 404 returned error can't find the container with id 8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf Oct 30 06:18:18.419839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:18.419813 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns"] Oct 30 06:18:19.383112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:19.383080 6171 generic.go:356] "Generic (PLEG): container finished" podID="6c2a5b71-eec1-4798-8f68-94d4d4046e6e" containerID="886bd73e625051ea0026cd3861d2aed88f04a32f71dc0ee2a4b72e7ecca9104e" exitCode=0 Oct 30 06:18:19.383461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:19.383153 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" event={"ID":"6c2a5b71-eec1-4798-8f68-94d4d4046e6e","Type":"ContainerDied","Data":"886bd73e625051ea0026cd3861d2aed88f04a32f71dc0ee2a4b72e7ecca9104e"} Oct 30 06:18:19.383461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:19.383210 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" event={"ID":"6c2a5b71-eec1-4798-8f68-94d4d4046e6e","Type":"ContainerStarted","Data":"8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf"} Oct 30 06:18:20.581159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.581133 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:18:20.587164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.587128 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:20.680539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.680454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\") pod \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " Oct 30 06:18:20.680716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.680672 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\") pod \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " Oct 30 06:18:20.680790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.680733 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\" (UniqueName: \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\") pod \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\" (UID: \"6c2a5b71-eec1-4798-8f68-94d4d4046e6e\") " Oct 30 06:18:20.680962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.680923 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca" pod "6c2a5b71-eec1-4798-8f68-94d4d4046e6e" (UID: "6c2a5b71-eec1-4798-8f68-94d4d4046e6e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:20.681546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.681516 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.684122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.684085 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca" pod "6c2a5b71-eec1-4798-8f68-94d4d4046e6e" (UID: "6c2a5b71-eec1-4798-8f68-94d4d4046e6e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:20.684244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.684161 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27" pod "6c2a5b71-eec1-4798-8f68-94d4d4046e6e" (UID: "6c2a5b71-eec1-4798-8f68-94d4d4046e6e"). InnerVolumeSpecName "kube-api-access-fcn27". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:20.782999 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.782976 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.783139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:20.783000 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcn27\" (UniqueName: \"kubernetes.io/projected/6c2a5b71-eec1-4798-8f68-94d4d4046e6e-kube-api-access-fcn27\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:21.459081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:21.455828 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" event={"ID":"6c2a5b71-eec1-4798-8f68-94d4d4046e6e","Type":"ContainerDied","Data":"8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf"} Oct 30 06:18:21.459081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:21.455909 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c5f8f8c6fc52a995d012e4433652f67d75c1d8a7a652419396f261dffb67ccf" Oct 30 06:18:21.459081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:21.455978 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns" Oct 30 06:18:26.387514 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:26.387477 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:18:29.214507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.214475 6171 scope.go:122] "RemoveContainer" containerID="18dbc86271dd813b01a71bf23c0ba66763ee9d68a0f87d43a413f618317b562b" Oct 30 06:18:29.229459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.229439 6171 scope.go:122] "RemoveContainer" containerID="e98fba4e0142483f3b598d4e1f26e42964357135e9acb6f99d2bfb5ed68494f9" Oct 30 06:18:29.243351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.243328 6171 scope.go:122] "RemoveContainer" containerID="0d44d8f8d209ee0e525c86c37874873c910d65dccfd11dd1fe11c2c3a34978f1" Oct 30 06:18:29.254393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.254372 6171 scope.go:122] "RemoveContainer" containerID="4c533104d3a5594dcd241c8c6ebef67043e62c23cf602352246e22d581900219" Oct 30 06:18:29.265005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.264979 6171 scope.go:122] "RemoveContainer" containerID="7c2ff3a9bf700081540154d4277a23421e82e7d40ec213e79735b6dd3113c14f" Oct 30 06:18:29.275388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:29.275370 6171 scope.go:122] "RemoveContainer" containerID="c7e214427a2923aa8e61f6398f1aa571bc38b0995273e9a30ffd3cc9c7e2c8ea" Oct 30 06:18:42.763310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:42.763274 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:18:42.764079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:42.763474 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" podUID="4dd37103-8329-4ecc-a977-a85065cc94ed" containerName="payload-processing" containerID="cri-o://5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901" gracePeriod=30 Oct 30 06:18:42.783163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:42.783133 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:18:42.783370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:42.783349 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" podUID="852fdab1-5b35-4059-bfcd-be832e68ad4f" containerName="payload-pre-processing" containerID="cri-o://ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada" gracePeriod=30 Oct 30 06:18:43.132322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.130942 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:18:43.188124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.181978 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\" (UniqueName: \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\") pod \"852fdab1-5b35-4059-bfcd-be832e68ad4f\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " Oct 30 06:18:43.188124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.182114 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\") pod \"852fdab1-5b35-4059-bfcd-be832e68ad4f\" (UID: \"852fdab1-5b35-4059-bfcd-be832e68ad4f\") " Oct 30 06:18:43.189002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.188949 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume" pod "852fdab1-5b35-4059-bfcd-be832e68ad4f" (UID: "852fdab1-5b35-4059-bfcd-be832e68ad4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:43.216003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.215948 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl" pod "852fdab1-5b35-4059-bfcd-be832e68ad4f" (UID: "852fdab1-5b35-4059-bfcd-be832e68ad4f"). InnerVolumeSpecName "kube-api-access-gj7rl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:43.265096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.263666 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:18:43.298118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.294761 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj7rl\" (UniqueName: \"kubernetes.io/projected/852fdab1-5b35-4059-bfcd-be832e68ad4f-kube-api-access-gj7rl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:43.298118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.294797 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/852fdab1-5b35-4059-bfcd-be832e68ad4f-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:43.397072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.396009 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\" (UniqueName: \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\") pod \"4dd37103-8329-4ecc-a977-a85065cc94ed\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " Oct 30 06:18:43.397072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.396308 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\") pod \"4dd37103-8329-4ecc-a977-a85065cc94ed\" (UID: \"4dd37103-8329-4ecc-a977-a85065cc94ed\") " Oct 30 06:18:43.431325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.418518 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume" pod "4dd37103-8329-4ecc-a977-a85065cc94ed" (UID: "4dd37103-8329-4ecc-a977-a85065cc94ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:43.431325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.419145 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd37103-8329-4ecc-a977-a85065cc94ed-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:43.478744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.478709 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm" pod "4dd37103-8329-4ecc-a977-a85065cc94ed" (UID: "4dd37103-8329-4ecc-a977-a85065cc94ed"). InnerVolumeSpecName "kube-api-access-5thkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:43.520532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.520498 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5thkm\" (UniqueName: \"kubernetes.io/projected/4dd37103-8329-4ecc-a977-a85065cc94ed-kube-api-access-5thkm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:43.584603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.584521 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns"] Oct 30 06:18:43.589838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.589804 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-2d4576-vx6ns"] Oct 30 06:18:43.692424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.692374 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c2a5b71-eec1-4798-8f68-94d4d4046e6e" path="/var/lib/kubelet/pods/6c2a5b71-eec1-4798-8f68-94d4d4046e6e/volumes" Oct 30 06:18:43.788895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.788852 6171 generic.go:356] "Generic (PLEG): container finished" podID="4dd37103-8329-4ecc-a977-a85065cc94ed" containerID="5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901" exitCode=0 Oct 30 06:18:43.789418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.788954 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" event={"ID":"4dd37103-8329-4ecc-a977-a85065cc94ed","Type":"ContainerDied","Data":"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901"} Oct 30 06:18:43.789418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.789004 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" event={"ID":"4dd37103-8329-4ecc-a977-a85065cc94ed","Type":"ContainerDied","Data":"e940262ef1e39c12b39a54422833954c97e9ab09c786cb53ca95e0cd63b80eb4"} Oct 30 06:18:43.789418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.789035 6171 scope.go:122] "RemoveContainer" containerID="5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901" Oct 30 06:18:43.789418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.789217 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9" Oct 30 06:18:43.793153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.793125 6171 generic.go:356] "Generic (PLEG): container finished" podID="852fdab1-5b35-4059-bfcd-be832e68ad4f" containerID="ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada" exitCode=0 Oct 30 06:18:43.793273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.793163 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" Oct 30 06:18:43.793273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.793164 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" event={"ID":"852fdab1-5b35-4059-bfcd-be832e68ad4f","Type":"ContainerDied","Data":"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada"} Oct 30 06:18:43.793273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.793218 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2" event={"ID":"852fdab1-5b35-4059-bfcd-be832e68ad4f","Type":"ContainerDied","Data":"e3abf719505ca70e12fe50554d193a9f796de1d65c612ce6b257e09681e215f1"} Oct 30 06:18:43.807280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.807259 6171 scope.go:122] "RemoveContainer" containerID="5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901" Oct 30 06:18:43.807574 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:43.807551 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901\": container with ID starting with 5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901 not found: ID does not exist" containerID="5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901" Oct 30 06:18:43.807647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.807581 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901"} err="failed to get container status \"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901\": rpc error: code = NotFound desc = could not find container \"5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901\": container with ID starting with 5195fc857a43c393240764217549f312a6bfb88bf21088b0ed5ef2cba3682901 not found: ID does not exist" Oct 30 06:18:43.807647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.807601 6171 scope.go:122] "RemoveContainer" containerID="ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada" Oct 30 06:18:43.820183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.820158 6171 scope.go:122] "RemoveContainer" containerID="ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada" Oct 30 06:18:43.820464 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:43.820434 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada\": container with ID starting with ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada not found: ID does not exist" containerID="ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada" Oct 30 06:18:43.820546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.820465 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada"} err="failed to get container status \"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada\": rpc error: code = NotFound desc = could not find container \"ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada\": container with ID starting with ba3e99935dd41eb9d4a26b970ffe664180dcff2cdc9b3a68a9434338e89c7ada not found: ID does not exist" Oct 30 06:18:43.857206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.857151 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:18:43.862396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.862374 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2d4576-75bb74d86c-cf5k2"] Oct 30 06:18:43.897055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.897010 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:18:43.901980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:43.901960 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2d4576-755f4cf4dd-7cng9"] Oct 30 06:18:44.034181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:44.034155 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5"] Oct 30 06:18:44.038876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:44.038853 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af2bc828-67vg5"] Oct 30 06:18:45.676258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:45.676226 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd37103-8329-4ecc-a977-a85065cc94ed" path="/var/lib/kubelet/pods/4dd37103-8329-4ecc-a977-a85065cc94ed/volumes" Oct 30 06:18:45.676737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:45.676721 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="852fdab1-5b35-4059-bfcd-be832e68ad4f" path="/var/lib/kubelet/pods/852fdab1-5b35-4059-bfcd-be832e68ad4f/volumes" Oct 30 06:18:45.677301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:45.677283 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c66167-575f-41f5-9652-406c7f5b12d0" path="/var/lib/kubelet/pods/a8c66167-575f-41f5-9652-406c7f5b12d0/volumes" Oct 30 06:18:51.169737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.169702 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw"] Oct 30 06:18:51.170657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170637 6171 cpu_manager.go:414] "containerMap: removing container" podUID="852fdab1-5b35-4059-bfcd-be832e68ad4f" containerName="payload-pre-processing" Oct 30 06:18:51.170718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170658 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="852fdab1-5b35-4059-bfcd-be832e68ad4f" containerName="payload-pre-processing" Oct 30 06:18:51.170718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170675 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6c2a5b71-eec1-4798-8f68-94d4d4046e6e" containerName="revoke-keys" Oct 30 06:18:51.170718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170681 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c2a5b71-eec1-4798-8f68-94d4d4046e6e" containerName="revoke-keys" Oct 30 06:18:51.170718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170689 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4dd37103-8329-4ecc-a977-a85065cc94ed" containerName="payload-processing" Oct 30 06:18:51.170718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170695 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd37103-8329-4ecc-a977-a85065cc94ed" containerName="payload-processing" Oct 30 06:18:51.170886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170851 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="852fdab1-5b35-4059-bfcd-be832e68ad4f" containerName="payload-pre-processing" Oct 30 06:18:51.170886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170859 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c2a5b71-eec1-4798-8f68-94d4d4046e6e" containerName="revoke-keys" Oct 30 06:18:51.170886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.170866 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd37103-8329-4ecc-a977-a85065cc94ed" containerName="payload-processing" Oct 30 06:18:51.175158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.175140 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.194336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.194311 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw"] Oct 30 06:18:51.208759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.208731 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:18:51.208870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.208737 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:18:51.289829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.289796 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.289982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.289958 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfbbg\" (UniqueName: \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.290032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.290003 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.391199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.391160 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.391392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.391293 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfbbg\" (UniqueName: \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.391392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.391342 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.392008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.391983 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.393795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.393769 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.416592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.416570 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfbbg\" (UniqueName: \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.492310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.492036 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:51.916067 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:18:51.914664 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc779e86e_4b15_4c5b_a924_f2adb695f1fe.slice/crio-0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9 WatchSource:0}: Error finding container 0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9: Status 404 returned error can't find the container with id 0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9 Oct 30 06:18:51.938069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.933703 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw"] Oct 30 06:18:51.956852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:51.956747 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" event={"ID":"c779e86e-4b15-4c5b-a924-f2adb695f1fe","Type":"ContainerStarted","Data":"0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9"} Oct 30 06:18:52.972183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:52.972147 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" event={"ID":"c779e86e-4b15-4c5b-a924-f2adb695f1fe","Type":"ContainerStarted","Data":"585320088a14033bc3c7c6675bc378d31366911c407b847d76e1e4e192107778"} Oct 30 06:18:53.028327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:53.028265 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" podStartSLOduration=2.028247723 podStartE2EDuration="2.028247723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:53.021429721 +0000 UTC m=+2791.461806915" watchObservedRunningTime="2026-10-30 06:18:53.028247723 +0000 UTC m=+2791.468624917" Oct 30 06:18:53.983076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:53.983014 6171 generic.go:356] "Generic (PLEG): container finished" podID="c779e86e-4b15-4c5b-a924-f2adb695f1fe" containerID="585320088a14033bc3c7c6675bc378d31366911c407b847d76e1e4e192107778" exitCode=0 Oct 30 06:18:53.983487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:53.983082 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" event={"ID":"c779e86e-4b15-4c5b-a924-f2adb695f1fe","Type":"ContainerDied","Data":"585320088a14033bc3c7c6675bc378d31366911c407b847d76e1e4e192107778"} Oct 30 06:18:54.048349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:54.048313 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6"] Oct 30 06:18:54.053113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:54.053089 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-790f8f63-nlzc6"] Oct 30 06:18:55.170127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.170099 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:55.222776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.222743 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\" (UniqueName: \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\") pod \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " Oct 30 06:18:55.222955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.222842 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\") pod \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " Oct 30 06:18:55.222955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.222889 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\") pod \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\" (UID: \"c779e86e-4b15-4c5b-a924-f2adb695f1fe\") " Oct 30 06:18:55.223365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.223342 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca" pod "c779e86e-4b15-4c5b-a924-f2adb695f1fe" (UID: "c779e86e-4b15-4c5b-a924-f2adb695f1fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:55.224025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.224000 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:55.226630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.226593 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg" pod "c779e86e-4b15-4c5b-a924-f2adb695f1fe" (UID: "c779e86e-4b15-4c5b-a924-f2adb695f1fe"). InnerVolumeSpecName "kube-api-access-gfbbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:55.226879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.226854 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca" pod "c779e86e-4b15-4c5b-a924-f2adb695f1fe" (UID: "c779e86e-4b15-4c5b-a924-f2adb695f1fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:55.325554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.325473 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfbbg\" (UniqueName: \"kubernetes.io/projected/c779e86e-4b15-4c5b-a924-f2adb695f1fe-kube-api-access-gfbbg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:55.325554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.325503 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c779e86e-4b15-4c5b-a924-f2adb695f1fe-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:18:55.701890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:55.701802 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406" path="/var/lib/kubelet/pods/e82eb8cb-48ca-4ce8-9bcf-223f7b9d2406/volumes" Oct 30 06:18:56.002667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:56.002636 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" Oct 30 06:18:56.002667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:56.002655 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw" event={"ID":"c779e86e-4b15-4c5b-a924-f2adb695f1fe","Type":"ContainerDied","Data":"0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9"} Oct 30 06:18:56.002868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:56.002708 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d7b21517ffe6bea70fdfea51eee4ed70146d5a834d9e0e407f471813ca85fc9" Oct 30 06:18:56.383005 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:18:56.382914 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:18:59.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.525807 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:18:59.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.527273 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c779e86e-4b15-4c5b-a924-f2adb695f1fe" containerName="revoke-keys" Oct 30 06:18:59.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.527296 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c779e86e-4b15-4c5b-a924-f2adb695f1fe" containerName="revoke-keys" Oct 30 06:18:59.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.527502 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c779e86e-4b15-4c5b-a924-f2adb695f1fe" containerName="revoke-keys" Oct 30 06:18:59.689600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.689567 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:18:59.697011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.689922 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.721951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.719665 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-f7ad21-dockercfg-8rzw9\"" Oct 30 06:18:59.721951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.721241 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-f7ad21\"" Oct 30 06:18:59.725448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.725422 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:18:59.731880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.731863 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:18:59.731989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.731940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.810600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.810534 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc4p9\" (UniqueName: \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.810747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.810720 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5j57\" (UniqueName: \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.810845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.810827 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.810926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.810868 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.911823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.911793 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc4p9\" (UniqueName: \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.911943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.911895 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5j57\" (UniqueName: \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.912006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.911944 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.912006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.911968 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.912689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.912664 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:18:59.912778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.912671 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.937350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.937318 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5j57\" (UniqueName: \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\") pod \"payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:18:59.937460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:18:59.937394 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc4p9\" (UniqueName: \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\") pod \"payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:19:00.036132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:00.036099 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:19:00.047811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:00.047779 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:19:00.383677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:00.383638 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:19:00.384400 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:00.384368 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aaf8aff_680a_4c29_a191_854acf6204a0.slice/crio-2a669fbe25f0f90d047b4bb3a4233d921a83c36184f8a0d25b27ff9a9a68c7c9 WatchSource:0}: Error finding container 2a669fbe25f0f90d047b4bb3a4233d921a83c36184f8a0d25b27ff9a9a68c7c9: Status 404 returned error can't find the container with id 2a669fbe25f0f90d047b4bb3a4233d921a83c36184f8a0d25b27ff9a9a68c7c9 Oct 30 06:19:00.590340 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:00.590312 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a25279c_bb5e_47fb_9b38_842b88ac29a4.slice/crio-bd8f66eb1d6ac87b3ef981e3003a993cf381f225d81e24475b34d83590ece827 WatchSource:0}: Error finding container bd8f66eb1d6ac87b3ef981e3003a993cf381f225d81e24475b34d83590ece827: Status 404 returned error can't find the container with id bd8f66eb1d6ac87b3ef981e3003a993cf381f225d81e24475b34d83590ece827 Oct 30 06:19:00.593926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:00.593896 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:19:01.061138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.061101 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" event={"ID":"3a25279c-bb5e-47fb-9b38-842b88ac29a4","Type":"ContainerStarted","Data":"e98d0d9acb5c9330f4908aba45201cf359615565c754d922bc3399faca9c72ec"} Oct 30 06:19:01.061328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.061165 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" event={"ID":"3a25279c-bb5e-47fb-9b38-842b88ac29a4","Type":"ContainerStarted","Data":"bd8f66eb1d6ac87b3ef981e3003a993cf381f225d81e24475b34d83590ece827"} Oct 30 06:19:01.061328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.061193 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:19:01.062904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.062877 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" event={"ID":"8aaf8aff-680a-4c29-a191-854acf6204a0","Type":"ContainerStarted","Data":"ea5283849e8a98a3c6102a5158678247ba65612a08013f895980974101f7c20e"} Oct 30 06:19:01.063087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.062933 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" event={"ID":"8aaf8aff-680a-4c29-a191-854acf6204a0","Type":"ContainerStarted","Data":"2a669fbe25f0f90d047b4bb3a4233d921a83c36184f8a0d25b27ff9a9a68c7c9"} Oct 30 06:19:01.063087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.062973 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:19:01.111726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.111679 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" podStartSLOduration=2.111662239 podStartE2EDuration="2.111662239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:01.107075517 +0000 UTC m=+2799.547452710" watchObservedRunningTime="2026-10-30 06:19:01.111662239 +0000 UTC m=+2799.552039428" Oct 30 06:19:01.158206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:01.158153 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" podStartSLOduration=2.158136609 podStartE2EDuration="2.158136609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:01.153445321 +0000 UTC m=+2799.593822514" watchObservedRunningTime="2026-10-30 06:19:01.158136609 +0000 UTC m=+2799.598513802" Oct 30 06:19:05.068821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:05.068783 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8"] Oct 30 06:19:05.082357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:05.082326 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96df05fe-q98v8"] Oct 30 06:19:05.677484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:05.677449 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df5cd092-7b59-4adc-bf47-fbdfd022f67e" path="/var/lib/kubelet/pods/df5cd092-7b59-4adc-bf47-fbdfd022f67e/volumes" Oct 30 06:19:12.075513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:12.075483 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:19:12.076035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:12.075564 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:19:13.032883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:13.032848 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j"] Oct 30 06:19:13.037859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:13.037828 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70f0ecbd-qks5j"] Oct 30 06:19:13.675964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:13.675917 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="912297fd-6a71-4a01-aa25-6be064421cbe" path="/var/lib/kubelet/pods/912297fd-6a71-4a01-aa25-6be064421cbe/volumes" Oct 30 06:19:16.043418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.043379 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv"] Oct 30 06:19:16.056983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.056937 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1805ecc1-8bkvv"] Oct 30 06:19:16.875511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.875484 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf"] Oct 30 06:19:16.883133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.883111 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:16.935621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.935586 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:19:16.935981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.935962 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:16.967670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:16.967615 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf"] Oct 30 06:19:17.012880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.012840 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.013082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.012959 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8rs4\" (UniqueName: \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.013082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.013029 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.114302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.114267 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.114710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.114324 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8rs4\" (UniqueName: \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.114710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.114378 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.114912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.114888 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.116899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.116875 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.140629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.140563 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8rs4\" (UniqueName: \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\") pod \"maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.200022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.199997 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:17.518574 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:17.518539 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb577654_0c8a_4647_a2ea_4e796c1bb89c.slice/crio-d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a WatchSource:0}: Error finding container d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a: Status 404 returned error can't find the container with id d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a Oct 30 06:19:17.521841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.521808 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf"] Oct 30 06:19:17.676216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:17.676179 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f0485c-7edf-45fd-a01d-f449f082f785" path="/var/lib/kubelet/pods/f8f0485c-7edf-45fd-a01d-f449f082f785/volumes" Oct 30 06:19:18.320186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:18.320148 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" event={"ID":"fb577654-0c8a-4647-a2ea-4e796c1bb89c","Type":"ContainerStarted","Data":"46bd08728ee1c18d66b2540dfb36bc94ec366e0cfa060ad1c6b325301fae67fd"} Oct 30 06:19:18.320613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:18.320205 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" event={"ID":"fb577654-0c8a-4647-a2ea-4e796c1bb89c","Type":"ContainerStarted","Data":"d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a"} Oct 30 06:19:18.369473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:18.369415 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" podStartSLOduration=2.369400871 podStartE2EDuration="2.369400871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:18.364227862 +0000 UTC m=+2816.804605055" watchObservedRunningTime="2026-10-30 06:19:18.369400871 +0000 UTC m=+2816.809778064" Oct 30 06:19:19.331873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:19.331839 6171 generic.go:356] "Generic (PLEG): container finished" podID="fb577654-0c8a-4647-a2ea-4e796c1bb89c" containerID="46bd08728ee1c18d66b2540dfb36bc94ec366e0cfa060ad1c6b325301fae67fd" exitCode=0 Oct 30 06:19:19.332376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:19.331919 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" event={"ID":"fb577654-0c8a-4647-a2ea-4e796c1bb89c","Type":"ContainerDied","Data":"46bd08728ee1c18d66b2540dfb36bc94ec366e0cfa060ad1c6b325301fae67fd"} Oct 30 06:19:20.538862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.538838 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:20.605182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.605082 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\") pod \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " Oct 30 06:19:20.605408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.605278 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\") pod \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " Oct 30 06:19:20.605408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.605356 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\" (UniqueName: \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\") pod \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\" (UID: \"fb577654-0c8a-4647-a2ea-4e796c1bb89c\") " Oct 30 06:19:20.605598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.605402 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca" pod "fb577654-0c8a-4647-a2ea-4e796c1bb89c" (UID: "fb577654-0c8a-4647-a2ea-4e796c1bb89c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:20.607909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.607885 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:20.624472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.624447 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca" pod "fb577654-0c8a-4647-a2ea-4e796c1bb89c" (UID: "fb577654-0c8a-4647-a2ea-4e796c1bb89c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:20.624614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.624592 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4" pod "fb577654-0c8a-4647-a2ea-4e796c1bb89c" (UID: "fb577654-0c8a-4647-a2ea-4e796c1bb89c"). InnerVolumeSpecName "kube-api-access-v8rs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:20.709867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.709837 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb577654-0c8a-4647-a2ea-4e796c1bb89c-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:20.709867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:20.709863 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8rs4\" (UniqueName: \"kubernetes.io/projected/fb577654-0c8a-4647-a2ea-4e796c1bb89c-kube-api-access-v8rs4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:21.352708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:21.352681 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" Oct 30 06:19:21.352897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:21.352702 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf" event={"ID":"fb577654-0c8a-4647-a2ea-4e796c1bb89c","Type":"ContainerDied","Data":"d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a"} Oct 30 06:19:21.352897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:21.352767 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53e01e8fa4b593df8addef738c655d2cbf513dd619f22d9ff695ca4e719835a" Oct 30 06:19:22.506141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:22.506105 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:19:24.034350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:24.034309 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m"] Oct 30 06:19:24.048057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:24.048021 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-acf0c297-wv27m"] Oct 30 06:19:25.677902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:25.677862 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1438f908-3dc2-45ff-9bbc-977bd0ed05a6" path="/var/lib/kubelet/pods/1438f908-3dc2-45ff-9bbc-977bd0ed05a6/volumes" Oct 30 06:19:26.382871 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:19:26.382834 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:19:26.488176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.488145 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq"] Oct 30 06:19:26.489065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.489034 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fb577654-0c8a-4647-a2ea-4e796c1bb89c" containerName="revoke-keys" Oct 30 06:19:26.489120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.489068 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb577654-0c8a-4647-a2ea-4e796c1bb89c" containerName="revoke-keys" Oct 30 06:19:26.489240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.489230 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb577654-0c8a-4647-a2ea-4e796c1bb89c" containerName="revoke-keys" Oct 30 06:19:26.494477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.494455 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.512664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.512641 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq"] Oct 30 06:19:26.520644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.520605 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:26.520813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.520764 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:19:26.622751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.622724 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.622924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.622784 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5znp8\" (UniqueName: \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.623033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.623013 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.724029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.724004 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.724518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.724138 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.724518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.724193 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5znp8\" (UniqueName: \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.724766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.724742 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.726622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.726597 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.749614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.749585 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5znp8\" (UniqueName: \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\") pod \"maas-api-revoke-sub-c4a6368c-r62mq\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:26.811175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:26.811155 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:27.127142 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:27.127114 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61fadbac_a73e_44e1_b498_8937a3eb2957.slice/crio-1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860 WatchSource:0}: Error finding container 1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860: Status 404 returned error can't find the container with id 1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860 Oct 30 06:19:27.127828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:27.127804 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq"] Oct 30 06:19:27.416094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:27.415982 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" event={"ID":"61fadbac-a73e-44e1-b498-8937a3eb2957","Type":"ContainerStarted","Data":"1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860"} Oct 30 06:19:28.426613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:28.426578 6171 generic.go:356] "Generic (PLEG): container finished" podID="61fadbac-a73e-44e1-b498-8937a3eb2957" containerID="946270ecf20b3bd3e0d9269dfd1ec4a30bad8f30c9d87c68c7fc26a6fc931d05" exitCode=0 Oct 30 06:19:28.427004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:28.426652 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" event={"ID":"61fadbac-a73e-44e1-b498-8937a3eb2957","Type":"ContainerDied","Data":"946270ecf20b3bd3e0d9269dfd1ec4a30bad8f30c9d87c68c7fc26a6fc931d05"} Oct 30 06:19:29.375836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.375803 6171 scope.go:122] "RemoveContainer" containerID="75b96b1b152b5c7d2893bdfcbfcf66c8be555239b63f80c5d069f76c65d2736a" Oct 30 06:19:29.388511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.388488 6171 scope.go:122] "RemoveContainer" containerID="bde3a8a9fffb7358d1205e2f01739c486d6dcc3b252272e5b0d64249a2b213e5" Oct 30 06:19:29.630247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.630186 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:29.712068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.703246 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\") pod \"61fadbac-a73e-44e1-b498-8937a3eb2957\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " Oct 30 06:19:29.712068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.703386 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\" (UniqueName: \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\") pod \"61fadbac-a73e-44e1-b498-8937a3eb2957\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " Oct 30 06:19:29.712068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.703478 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\") pod \"61fadbac-a73e-44e1-b498-8937a3eb2957\" (UID: \"61fadbac-a73e-44e1-b498-8937a3eb2957\") " Oct 30 06:19:29.716631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.716595 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca" pod "61fadbac-a73e-44e1-b498-8937a3eb2957" (UID: "61fadbac-a73e-44e1-b498-8937a3eb2957"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:29.731151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.730533 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca" pod "61fadbac-a73e-44e1-b498-8937a3eb2957" (UID: "61fadbac-a73e-44e1-b498-8937a3eb2957"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:29.758157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.756286 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8" pod "61fadbac-a73e-44e1-b498-8937a3eb2957" (UID: "61fadbac-a73e-44e1-b498-8937a3eb2957"). InnerVolumeSpecName "kube-api-access-5znp8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:29.813974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.813938 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:29.814177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.813978 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61fadbac-a73e-44e1-b498-8937a3eb2957-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:29.814177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:29.813995 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5znp8\" (UniqueName: \"kubernetes.io/projected/61fadbac-a73e-44e1-b498-8937a3eb2957-kube-api-access-5znp8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:30.448914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:30.448881 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" Oct 30 06:19:30.449122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:30.448880 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq" event={"ID":"61fadbac-a73e-44e1-b498-8937a3eb2957","Type":"ContainerDied","Data":"1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860"} Oct 30 06:19:30.449122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:30.449027 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4604b59adb90d64c40341287ac4a04f004671e31c9d0ac4f0cd851848de860" Oct 30 06:19:34.118356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:34.118323 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw"] Oct 30 06:19:34.159408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:34.159369 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-cfe14f-5r4vw"] Oct 30 06:19:35.768023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:35.767987 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c779e86e-4b15-4c5b-a924-f2adb695f1fe" path="/var/lib/kubelet/pods/c779e86e-4b15-4c5b-a924-f2adb695f1fe/volumes" Oct 30 06:19:36.879184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.879148 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz"] Oct 30 06:19:36.880601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.880580 6171 cpu_manager.go:414] "containerMap: removing container" podUID="61fadbac-a73e-44e1-b498-8937a3eb2957" containerName="revoke-keys" Oct 30 06:19:36.880709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.880603 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61fadbac-a73e-44e1-b498-8937a3eb2957" containerName="revoke-keys" Oct 30 06:19:36.880825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.880814 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="61fadbac-a73e-44e1-b498-8937a3eb2957" containerName="revoke-keys" Oct 30 06:19:36.886941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.886914 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:36.902627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.902600 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz"] Oct 30 06:19:36.908683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.908659 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:36.908856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:36.908803 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:19:37.004649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.004613 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wxpw\" (UniqueName: \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.004793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.004770 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.004961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.004946 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.106677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.106644 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.106862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.106728 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wxpw\" (UniqueName: \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.106862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.106810 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.107463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.107439 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.109232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.109210 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.131919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.131855 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wxpw\" (UniqueName: \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\") pod \"maas-api-revoke-sub-e9a9d6b1-q97pz\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.206431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.206408 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:37.518339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:37.518309 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz"] Oct 30 06:19:37.519858 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:37.519832 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372310df_6b38_4894_a936_3c64a4f086ef.slice/crio-1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258 WatchSource:0}: Error finding container 1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258: Status 404 returned error can't find the container with id 1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258 Oct 30 06:19:38.529681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:38.529648 6171 generic.go:356] "Generic (PLEG): container finished" podID="372310df-6b38-4894-a936-3c64a4f086ef" containerID="0d73fe1258fc1b89e961fa36d9ed1a221731d04f8c4e72b8e094583a835a4b2d" exitCode=0 Oct 30 06:19:38.530166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:38.529692 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" event={"ID":"372310df-6b38-4894-a936-3c64a4f086ef","Type":"ContainerDied","Data":"0d73fe1258fc1b89e961fa36d9ed1a221731d04f8c4e72b8e094583a835a4b2d"} Oct 30 06:19:38.530166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:38.529751 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" event={"ID":"372310df-6b38-4894-a936-3c64a4f086ef","Type":"ContainerStarted","Data":"1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258"} Oct 30 06:19:39.971292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:39.971265 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:40.027399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.027367 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\") pod \"372310df-6b38-4894-a936-3c64a4f086ef\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " Oct 30 06:19:40.027594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.027460 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\") pod \"372310df-6b38-4894-a936-3c64a4f086ef\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " Oct 30 06:19:40.027594 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.027568 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\" (UniqueName: \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\") pod \"372310df-6b38-4894-a936-3c64a4f086ef\" (UID: \"372310df-6b38-4894-a936-3c64a4f086ef\") " Oct 30 06:19:40.027785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.027757 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca" pod "372310df-6b38-4894-a936-3c64a4f086ef" (UID: "372310df-6b38-4894-a936-3c64a4f086ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:40.028443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.028416 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:40.030912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.030886 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca" pod "372310df-6b38-4894-a936-3c64a4f086ef" (UID: "372310df-6b38-4894-a936-3c64a4f086ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:40.031106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.031079 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw" pod "372310df-6b38-4894-a936-3c64a4f086ef" (UID: "372310df-6b38-4894-a936-3c64a4f086ef"). InnerVolumeSpecName "kube-api-access-5wxpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:40.129567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.129520 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372310df-6b38-4894-a936-3c64a4f086ef-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:40.129567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.129561 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wxpw\" (UniqueName: \"kubernetes.io/projected/372310df-6b38-4894-a936-3c64a4f086ef-kube-api-access-5wxpw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:40.551866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.551837 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" Oct 30 06:19:40.551866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.551866 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz" event={"ID":"372310df-6b38-4894-a936-3c64a4f086ef","Type":"ContainerDied","Data":"1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258"} Oct 30 06:19:40.552118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:40.551913 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e5333fbb98d5e085527e526b5805fbdb9a78cab5af55c06254ad2bcd0f9b258" Oct 30 06:19:41.260242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.260207 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:19:41.260752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.260522 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" containerID="cri-o://4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e" gracePeriod=30 Oct 30 06:19:41.827102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.827067 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw"] Oct 30 06:19:41.828153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.828130 6171 cpu_manager.go:414] "containerMap: removing container" podUID="372310df-6b38-4894-a936-3c64a4f086ef" containerName="revoke-keys" Oct 30 06:19:41.828153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.828149 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372310df-6b38-4894-a936-3c64a4f086ef" containerName="revoke-keys" Oct 30 06:19:41.828386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.828327 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="372310df-6b38-4894-a936-3c64a4f086ef" containerName="revoke-keys" Oct 30 06:19:41.832526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.832506 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.833015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.832991 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:19:41.851906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.851874 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw"] Oct 30 06:19:41.860556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.860528 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:41.860741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.860580 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:19:41.862618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.862594 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:19:41.873617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.873595 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9rw2\" (UniqueName: \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.873802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.873790 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.873851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.873822 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.975244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.975202 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9rw2\" (UniqueName: \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.975789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.975496 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.975789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.975554 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.976393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.976368 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:41.985380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:41.985356 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:42.001928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:42.001908 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9rw2\" (UniqueName: \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:42.148642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:42.148556 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:42.478746 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:42.478718 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cc76f55_5777_4bb7_9e91_b2c4469775a0.slice/crio-de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15 WatchSource:0}: Error finding container de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15: Status 404 returned error can't find the container with id de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15 Oct 30 06:19:42.479792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:42.479771 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw"] Oct 30 06:19:42.726329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:42.726127 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" event={"ID":"4cc76f55-5777-4bb7-9e91-b2c4469775a0","Type":"ContainerStarted","Data":"de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15"} Oct 30 06:19:43.737549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:43.737515 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" event={"ID":"4cc76f55-5777-4bb7-9e91-b2c4469775a0","Type":"ContainerStarted","Data":"189c968c35401fbc452e73b26657f8c962a4a013c098a7ae3bb90379091b77d8"} Oct 30 06:19:43.799700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:43.799647 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" podStartSLOduration=2.79963427 podStartE2EDuration="2.79963427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:43.79616698 +0000 UTC m=+2842.236544173" watchObservedRunningTime="2026-10-30 06:19:43.79963427 +0000 UTC m=+2842.240011461" Oct 30 06:19:44.749088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:44.749021 6171 generic.go:356] "Generic (PLEG): container finished" podID="4cc76f55-5777-4bb7-9e91-b2c4469775a0" containerID="189c968c35401fbc452e73b26657f8c962a4a013c098a7ae3bb90379091b77d8" exitCode=0 Oct 30 06:19:44.749586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:44.749106 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" event={"ID":"4cc76f55-5777-4bb7-9e91-b2c4469775a0","Type":"ContainerDied","Data":"189c968c35401fbc452e73b26657f8c962a4a013c098a7ae3bb90379091b77d8"} Oct 30 06:19:45.478080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.478030 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:19:45.483627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.483604 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.506539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.506511 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:19:45.513206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.513157 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-5deb36-openshift-default-dockercfg-7vlwt\"" Oct 30 06:19:45.536836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.536804 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.536880 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncm45\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537021 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537097 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537164 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537345 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537421 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.537748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.537489 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639116 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639166 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639252 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639280 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639365 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639453 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639500 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncm45\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639537 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639690 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639742 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639773 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639860 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.639992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.639978 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.641803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.641784 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.641911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.641852 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.664808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.664783 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.665109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.665005 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncm45\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:45.800417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:45.800379 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:46.037728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.037664 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:46.148218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.148189 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\") pod \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " Oct 30 06:19:46.148399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.148294 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\") pod \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " Oct 30 06:19:46.148470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.148454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\" (UniqueName: \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\") pod \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\" (UID: \"4cc76f55-5777-4bb7-9e91-b2c4469775a0\") " Oct 30 06:19:46.148637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.148605 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca" pod "4cc76f55-5777-4bb7-9e91-b2c4469775a0" (UID: "4cc76f55-5777-4bb7-9e91-b2c4469775a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:46.149038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.149021 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.150465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.150444 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca" pod "4cc76f55-5777-4bb7-9e91-b2c4469775a0" (UID: "4cc76f55-5777-4bb7-9e91-b2c4469775a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:46.150559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.150533 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2" pod "4cc76f55-5777-4bb7-9e91-b2c4469775a0" (UID: "4cc76f55-5777-4bb7-9e91-b2c4469775a0"). InnerVolumeSpecName "kube-api-access-v9rw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:46.159084 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:46.159022 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7244299_ce2c_4c21_bd1b_fded0b5c174f.slice/crio-49c9c51e92bacd35d02daa581b154f36e627b500a771a928c217bd225524eb62 WatchSource:0}: Error finding container 49c9c51e92bacd35d02daa581b154f36e627b500a771a928c217bd225524eb62: Status 404 returned error can't find the container with id 49c9c51e92bacd35d02daa581b154f36e627b500a771a928c217bd225524eb62 Oct 30 06:19:46.160490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.160471 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:19:46.162013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.161979 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:46.162124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.162064 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:46.162124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.162104 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:46.250076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.250034 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cc76f55-5777-4bb7-9e91-b2c4469775a0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.250192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.250079 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9rw2\" (UniqueName: \"kubernetes.io/projected/4cc76f55-5777-4bb7-9e91-b2c4469775a0-kube-api-access-v9rw2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.551624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.551596 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:19:46.659463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659428 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659641 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659726 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659791 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659838 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659873 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.659995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.659964 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.660096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.660076 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.660237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.660204 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\") pod \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\" (UID: \"e3c881bc-70da-4e83-be3b-d20fee1b1ae9\") " Oct 30 06:19:46.661125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.661086 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:46.661339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.661315 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:46.661763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.661740 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:46.662785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.662750 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:46.663097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.663073 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:46.664892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.664864 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:46.665184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.665159 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:19:46.667261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.667218 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:19:46.668827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.668204 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q" pod "e3c881bc-70da-4e83-be3b-d20fee1b1ae9" (UID: "e3c881bc-70da-4e83-be3b-d20fee1b1ae9"). InnerVolumeSpecName "kube-api-access-qts8q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:46.762338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762304 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qts8q\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-kube-api-access-qts8q\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762336 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762348 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762357 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762369 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762383 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762391 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762400 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.762618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.762407 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e3c881bc-70da-4e83-be3b-d20fee1b1ae9-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:46.770013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.769990 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" Oct 30 06:19:46.770191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.770027 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw" event={"ID":"4cc76f55-5777-4bb7-9e91-b2c4469775a0","Type":"ContainerDied","Data":"de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15"} Oct 30 06:19:46.770191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.770089 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de0f42c89f105a09be913a2c909bea53bb54d7084ea3276348238c513d42cd15" Oct 30 06:19:46.771890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.771868 6171 generic.go:356] "Generic (PLEG): container finished" podID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerID="4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e" exitCode=0 Oct 30 06:19:46.772021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.771922 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" Oct 30 06:19:46.772021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.771938 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" event={"ID":"e3c881bc-70da-4e83-be3b-d20fee1b1ae9","Type":"ContainerDied","Data":"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e"} Oct 30 06:19:46.772021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.772001 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx" event={"ID":"e3c881bc-70da-4e83-be3b-d20fee1b1ae9","Type":"ContainerDied","Data":"c74d3a16352f21ccf4cdef178853cf47f0fbc4b129e0e2a11de01de8f1ab8d36"} Oct 30 06:19:46.772208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.772031 6171 scope.go:122] "RemoveContainer" containerID="4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e" Oct 30 06:19:46.773985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.773959 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" event={"ID":"c7244299-ce2c-4c21-bd1b-fded0b5c174f","Type":"ContainerStarted","Data":"d8bc8392161b62ae3f512013c0100fd97b527c42c7c4df16d4098c26381d8392"} Oct 30 06:19:46.774137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.774011 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" event={"ID":"c7244299-ce2c-4c21-bd1b-fded0b5c174f","Type":"ContainerStarted","Data":"49c9c51e92bacd35d02daa581b154f36e627b500a771a928c217bd225524eb62"} Oct 30 06:19:46.786548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.786368 6171 scope.go:122] "RemoveContainer" containerID="4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e" Oct 30 06:19:46.786612 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:19:46.786594 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e\": container with ID starting with 4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e not found: ID does not exist" containerID="4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e" Oct 30 06:19:46.786657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.786620 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e"} err="failed to get container status \"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e\": rpc error: code = NotFound desc = could not find container \"4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e\": container with ID starting with 4cea9a2dfc66d138f386318becf3d85b076ea6057be21e97884b7f00a470442e not found: ID does not exist" Oct 30 06:19:46.810242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.810223 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:46.833445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.833415 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" podUID="9272efda-02aa-4f36-bbc5-5079c6019f76" containerName="payload-processing" containerID="cri-o://1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede" gracePeriod=25 Oct 30 06:19:46.835458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.835436 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:19:46.840342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.840319 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-cfe14f-openshift-default-746f4c9457-zg5zx"] Oct 30 06:19:46.862913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.862880 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" podUID="b664e835-8fa6-47c9-9791-c3cf95303019" containerName="payload-pre-processing" containerID="cri-o://24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd" gracePeriod=25 Oct 30 06:19:46.883843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:46.883782 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podStartSLOduration=1.883764454 podStartE2EDuration="1.883764454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:46.88168945 +0000 UTC m=+2845.322066643" watchObservedRunningTime="2026-10-30 06:19:46.883764454 +0000 UTC m=+2845.324141652" Oct 30 06:19:47.224954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.224923 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:19:47.247530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.247504 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:19:47.277109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.277081 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\" (UniqueName: \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\") pod \"9272efda-02aa-4f36-bbc5-5079c6019f76\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " Oct 30 06:19:47.277265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.277208 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\") pod \"9272efda-02aa-4f36-bbc5-5079c6019f76\" (UID: \"9272efda-02aa-4f36-bbc5-5079c6019f76\") " Oct 30 06:19:47.278239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.277564 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume" pod "9272efda-02aa-4f36-bbc5-5079c6019f76" (UID: "9272efda-02aa-4f36-bbc5-5079c6019f76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:47.278239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.278206 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9272efda-02aa-4f36-bbc5-5079c6019f76-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:47.279538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.279513 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48" pod "9272efda-02aa-4f36-bbc5-5079c6019f76" (UID: "9272efda-02aa-4f36-bbc5-5079c6019f76"). InnerVolumeSpecName "kube-api-access-6gd48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:47.379264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.379232 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\" (UniqueName: \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\") pod \"b664e835-8fa6-47c9-9791-c3cf95303019\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " Oct 30 06:19:47.379457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.379400 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\") pod \"b664e835-8fa6-47c9-9791-c3cf95303019\" (UID: \"b664e835-8fa6-47c9-9791-c3cf95303019\") " Oct 30 06:19:47.379817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.379788 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume" pod "b664e835-8fa6-47c9-9791-c3cf95303019" (UID: "b664e835-8fa6-47c9-9791-c3cf95303019"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:47.379991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.379969 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gd48\" (UniqueName: \"kubernetes.io/projected/9272efda-02aa-4f36-bbc5-5079c6019f76-kube-api-access-6gd48\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:47.388709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.388660 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8" pod "b664e835-8fa6-47c9-9791-c3cf95303019" (UID: "b664e835-8fa6-47c9-9791-c3cf95303019"). InnerVolumeSpecName "kube-api-access-fjpn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:47.417660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.417623 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:19:47.419033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419004 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b664e835-8fa6-47c9-9791-c3cf95303019" containerName="payload-pre-processing" Oct 30 06:19:47.419033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419030 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b664e835-8fa6-47c9-9791-c3cf95303019" containerName="payload-pre-processing" Oct 30 06:19:47.419033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419073 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9272efda-02aa-4f36-bbc5-5079c6019f76" containerName="payload-processing" Oct 30 06:19:47.419319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419081 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9272efda-02aa-4f36-bbc5-5079c6019f76" containerName="payload-processing" Oct 30 06:19:47.419319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419108 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" Oct 30 06:19:47.419319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419114 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" Oct 30 06:19:47.419319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419163 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4cc76f55-5777-4bb7-9e91-b2c4469775a0" containerName="revoke-keys" Oct 30 06:19:47.419319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419170 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cc76f55-5777-4bb7-9e91-b2c4469775a0" containerName="revoke-keys" Oct 30 06:19:47.419572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419406 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b664e835-8fa6-47c9-9791-c3cf95303019" containerName="payload-pre-processing" Oct 30 06:19:47.419572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419422 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9272efda-02aa-4f36-bbc5-5079c6019f76" containerName="payload-processing" Oct 30 06:19:47.419572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419438 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" containerName="istio-proxy" Oct 30 06:19:47.419572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.419456 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cc76f55-5777-4bb7-9e91-b2c4469775a0" containerName="revoke-keys" Oct 30 06:19:47.423774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.423753 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.431466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.431439 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:19:47.441436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.441418 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-25d881-openshift-default-dockercfg-8djl9\"" Oct 30 06:19:47.505187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.505138 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.505463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.505436 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.505765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.505744 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.505931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.505914 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509384 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509487 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f498j\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509619 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509736 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509790 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509892 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b664e835-8fa6-47c9-9791-c3cf95303019-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:47.510098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.509914 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjpn8\" (UniqueName: \"kubernetes.io/projected/b664e835-8fa6-47c9-9791-c3cf95303019-kube-api-access-fjpn8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:47.612128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612083 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612216 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612269 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612328 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612482 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612539 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612562 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612591 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612639 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612664 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f498j\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612825 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.612948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.612871 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.613582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.613493 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.615402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.615379 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.615598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.615579 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.638296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.638276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.638402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.638294 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f498j\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\") pod \"e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.676241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.676216 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3c881bc-70da-4e83-be3b-d20fee1b1ae9" path="/var/lib/kubelet/pods/e3c881bc-70da-4e83-be3b-d20fee1b1ae9/volumes" Oct 30 06:19:47.745287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.745253 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:47.817619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.816934 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:47.817619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.817010 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:47.828386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.828359 6171 generic.go:356] "Generic (PLEG): container finished" podID="b664e835-8fa6-47c9-9791-c3cf95303019" containerID="24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd" exitCode=0 Oct 30 06:19:47.828554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.828419 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" Oct 30 06:19:47.828554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.828454 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" event={"ID":"b664e835-8fa6-47c9-9791-c3cf95303019","Type":"ContainerDied","Data":"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd"} Oct 30 06:19:47.828554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.828516 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p" event={"ID":"b664e835-8fa6-47c9-9791-c3cf95303019","Type":"ContainerDied","Data":"e1d5bc33672260e2d53b6bc7002eaad10a1a6b19f5eee16ed4b0c7258911c4b9"} Oct 30 06:19:47.828554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.828541 6171 scope.go:122] "RemoveContainer" containerID="24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd" Oct 30 06:19:47.833107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.833083 6171 generic.go:356] "Generic (PLEG): container finished" podID="9272efda-02aa-4f36-bbc5-5079c6019f76" containerID="1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede" exitCode=0 Oct 30 06:19:47.833359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.833132 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" Oct 30 06:19:47.833564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.833144 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" event={"ID":"9272efda-02aa-4f36-bbc5-5079c6019f76","Type":"ContainerDied","Data":"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede"} Oct 30 06:19:47.834336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.834314 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn" event={"ID":"9272efda-02aa-4f36-bbc5-5079c6019f76","Type":"ContainerDied","Data":"d6bc6ccef419d4779c4815d615444b753e6a0a0a41304a525588137db03044db"} Oct 30 06:19:47.884965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.876731 6171 scope.go:122] "RemoveContainer" containerID="24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd" Oct 30 06:19:47.907087 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:19:47.905617 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd\": container with ID starting with 24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd not found: ID does not exist" containerID="24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd" Oct 30 06:19:47.907087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.905665 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd"} err="failed to get container status \"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd\": rpc error: code = NotFound desc = could not find container \"24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd\": container with ID starting with 24d3c77e28c9fc542269fcf07daa16fd1e810cc911d3de00eab6310cbd17e7dd not found: ID does not exist" Oct 30 06:19:47.907087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.905692 6171 scope.go:122] "RemoveContainer" containerID="1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede" Oct 30 06:19:47.948174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.942491 6171 scope.go:122] "RemoveContainer" containerID="1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede" Oct 30 06:19:47.948174 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:19:47.943378 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede\": container with ID starting with 1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede not found: ID does not exist" containerID="1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede" Oct 30 06:19:47.948174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.943430 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede"} err="failed to get container status \"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede\": rpc error: code = NotFound desc = could not find container \"1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede\": container with ID starting with 1193a3e286d5977e5df2139e8323253f3b24346d0751e6c5732ba3268215cede not found: ID does not exist" Oct 30 06:19:47.949948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.949890 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:19:47.956961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:47.956940 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5deb36-6cc8486645-t2hdn"] Oct 30 06:19:48.270535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.270431 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:19:48.290649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.290608 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:48.295408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.295374 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:48.295606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.295585 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:48.307869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.307840 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:19:48.307990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.307923 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5c9868557f-xxb4p"] Oct 30 06:19:48.524216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.524111 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:19:48.530455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.530430 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.551702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.551677 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-d51cd8-openshift-default-dockercfg-gdp6w\"" Oct 30 06:19:48.558814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.558785 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628381 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628445 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628546 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628581 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628668 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628715 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628788 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628817 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.644862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.628861 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sh2m\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.730185 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731062 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731166 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731224 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731394 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731505 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731538 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.731572 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sh2m\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.732821 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.733183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.733854 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.734151 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.734316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.736096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.735289 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.741875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.741848 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.757358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.757335 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.757849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.757828 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sh2m\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.815120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.814456 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:48.815120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.814524 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:48.886428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.886393 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:48.954181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.954138 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw"] Oct 30 06:19:48.964622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.964580 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:48.971117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.971092 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw"] Oct 30 06:19:49.001130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.987666 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:49.001130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:48.988364 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:19:49.045952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.037272 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.045952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.037391 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.045952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.037562 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9wq5\" (UniqueName: \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.045952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.042807 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" event={"ID":"5ed69623-150a-4bc7-864b-b1bd7e822605","Type":"ContainerStarted","Data":"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4"} Oct 30 06:19:49.045952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.042860 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" event={"ID":"5ed69623-150a-4bc7-864b-b1bd7e822605","Type":"ContainerStarted","Data":"40db64ce3ae393663982f9bdf807f8bbd5113a8d76f2aae153d9d3cefeb7bacb"} Oct 30 06:19:49.124678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.124559 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podStartSLOduration=2.124541279 podStartE2EDuration="2.124541279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:49.120479548 +0000 UTC m=+2847.560856742" watchObservedRunningTime="2026-10-30 06:19:49.124541279 +0000 UTC m=+2847.564918473" Oct 30 06:19:49.140382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.140351 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9wq5\" (UniqueName: \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.141565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.141543 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.141853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.141833 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.143937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.143773 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.147133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.146355 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.175752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.175699 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9wq5\" (UniqueName: \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.360574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.360545 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:49.411724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.411688 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:19:49.412498 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:49.412469 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09d31732_0dc7_4ae4_a935_9f48ead7bd24.slice/crio-864dd5a81f946301e390741589fb21eb9af917c201c2c1693b683a311af35ccf WatchSource:0}: Error finding container 864dd5a81f946301e390741589fb21eb9af917c201c2c1693b683a311af35ccf: Status 404 returned error can't find the container with id 864dd5a81f946301e390741589fb21eb9af917c201c2c1693b683a311af35ccf Oct 30 06:19:49.416033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.415998 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:49.416151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.416122 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:49.416202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.416181 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:19:49.758418 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:19:49.758152 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1155157_f3bc_40c7_b0fb_1922161e9196.slice/crio-bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0 WatchSource:0}: Error finding container bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0: Status 404 returned error can't find the container with id bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0 Oct 30 06:19:49.770691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.767972 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:49.770691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.768018 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:49.846068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.831708 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:49.846068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.831766 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:49.846068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.845452 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9272efda-02aa-4f36-bbc5-5079c6019f76" path="/var/lib/kubelet/pods/9272efda-02aa-4f36-bbc5-5079c6019f76/volumes" Oct 30 06:19:49.846401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.846240 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b664e835-8fa6-47c9-9791-c3cf95303019" path="/var/lib/kubelet/pods/b664e835-8fa6-47c9-9791-c3cf95303019/volumes" Oct 30 06:19:49.846735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.846713 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw"] Oct 30 06:19:49.846857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:49.846781 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:50.073869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.073834 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" event={"ID":"09d31732-0dc7-4ae4-a935-9f48ead7bd24","Type":"ContainerStarted","Data":"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0"} Oct 30 06:19:50.074312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.073888 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" event={"ID":"09d31732-0dc7-4ae4-a935-9f48ead7bd24","Type":"ContainerStarted","Data":"864dd5a81f946301e390741589fb21eb9af917c201c2c1693b683a311af35ccf"} Oct 30 06:19:50.077291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.077264 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" event={"ID":"b1155157-f3bc-40c7-b0fb-1922161e9196","Type":"ContainerStarted","Data":"bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0"} Oct 30 06:19:50.144093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.140942 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podStartSLOduration=2.140921835 podStartE2EDuration="2.140921835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:50.126003252 +0000 UTC m=+2848.566380456" watchObservedRunningTime="2026-10-30 06:19:50.140921835 +0000 UTC m=+2848.581299028" Oct 30 06:19:50.758711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.758674 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:50.758907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.758730 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:50.864793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.864177 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:50.864793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.864239 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:50.902028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.901991 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:50.903268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.903239 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:50.903482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:50.903456 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:51.166771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.166739 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" event={"ID":"b1155157-f3bc-40c7-b0fb-1922161e9196","Type":"ContainerStarted","Data":"5c151b30e04348c228d7c50334e386d14a63728ae89255349fb49dedcfa39462"} Oct 30 06:19:51.759526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.759475 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:51.759722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.759540 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:51.811069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.810749 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:51.811069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.810825 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:51.900068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.889407 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:51.900068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:51.889468 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:52.248900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.248862 6171 generic.go:356] "Generic (PLEG): container finished" podID="b1155157-f3bc-40c7-b0fb-1922161e9196" containerID="5c151b30e04348c228d7c50334e386d14a63728ae89255349fb49dedcfa39462" exitCode=0 Oct 30 06:19:52.259095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.251105 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" event={"ID":"b1155157-f3bc-40c7-b0fb-1922161e9196","Type":"ContainerDied","Data":"5c151b30e04348c228d7c50334e386d14a63728ae89255349fb49dedcfa39462"} Oct 30 06:19:52.756569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.756515 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:52.756946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.756900 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:52.845215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.833884 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" start-of-body= Oct 30 06:19:52.845215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:52.844144 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.130:15021/healthz/ready\": dial tcp 10.133.1.130:15021: connect: connection refused" Oct 30 06:19:53.048076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.046533 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:53.048076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.046592 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:53.757115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.756704 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:53.757115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.756748 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:53.846216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.845217 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:53.847232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.847172 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:53.847600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.847555 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:19:53.868199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.868180 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:53.927262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.927163 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:53.927524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:53.927494 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:54.036888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.036832 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\") pod \"b1155157-f3bc-40c7-b0fb-1922161e9196\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " Oct 30 06:19:54.037155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.036951 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\") pod \"b1155157-f3bc-40c7-b0fb-1922161e9196\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " Oct 30 06:19:54.037155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.037035 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\" (UniqueName: \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\") pod \"b1155157-f3bc-40c7-b0fb-1922161e9196\" (UID: \"b1155157-f3bc-40c7-b0fb-1922161e9196\") " Oct 30 06:19:54.039949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.039921 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca" pod "b1155157-f3bc-40c7-b0fb-1922161e9196" (UID: "b1155157-f3bc-40c7-b0fb-1922161e9196"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:54.042560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.042535 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca" pod "b1155157-f3bc-40c7-b0fb-1922161e9196" (UID: "b1155157-f3bc-40c7-b0fb-1922161e9196"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:54.044836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.044807 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5" pod "b1155157-f3bc-40c7-b0fb-1922161e9196" (UID: "b1155157-f3bc-40c7-b0fb-1922161e9196"). InnerVolumeSpecName "kube-api-access-j9wq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:54.141079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.141024 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:54.141079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.141081 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b1155157-f3bc-40c7-b0fb-1922161e9196-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:54.141360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.141095 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9wq5\" (UniqueName: \"kubernetes.io/projected/b1155157-f3bc-40c7-b0fb-1922161e9196-kube-api-access-j9wq5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:19:54.270327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.270125 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" Oct 30 06:19:54.270327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.270155 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw" event={"ID":"b1155157-f3bc-40c7-b0fb-1922161e9196","Type":"ContainerDied","Data":"bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0"} Oct 30 06:19:54.270327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.270217 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bef024b4edfdaa4fd4edd91a899f621e76db3d9f593097f8a6e315125c0d4db0" Oct 30 06:19:54.756879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.756846 6171 patch_prober.go:29] interesting pod/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.757071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.756892 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.131:15021/healthz/ready\": dial tcp 10.133.1.131:15021: connect: connection refused" Oct 30 06:19:54.901149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.900400 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:54.901149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:54.900459 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:55.901097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:55.897677 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" start-of-body= Oct 30 06:19:55.901097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:55.897736 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.132:15021/healthz/ready\": dial tcp 10.133.1.132:15021: connect: connection refused" Oct 30 06:19:55.909068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:55.907313 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:55.909068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:55.909022 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:55.915577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:55.915556 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:19:56.404691 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:19:56.404657 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:19:56.894585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:56.894552 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:56.895059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:56.895024 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:56.895946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:56.895927 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:19:59.367459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:59.367427 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:19:59.419547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:19:59.419494 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:20:04.369161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:04.369124 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" containerID="cri-o://3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb" gracePeriod=25 Oct 30 06:20:04.420404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:04.420323 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" containerID="cri-o://2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df" gracePeriod=25 Oct 30 06:20:09.678912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:09.678864 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:09.679418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:09.678918 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:15.217704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:15.217669 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:20:15.218191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:15.217924 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" containerID="cri-o://0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f" gracePeriod=30 Oct 30 06:20:19.678788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:19.678753 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:19.679242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:19.678757 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:20.533265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.533237 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:20:20.586282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586252 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586311 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586403 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586469 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586548 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586614 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586645 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:20.586871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586756 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.586871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586709 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.586871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586825 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.587065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\") pod \"0708855f-2a72-4f54-8ee5-2df4c073be69\" (UID: \"0708855f-2a72-4f54-8ee5-2df4c073be69\") " Oct 30 06:20:20.587065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.586957 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.587187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.587060 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.587318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.587294 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.587969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.587942 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.588117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.587973 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.588117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.587987 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.588117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.588001 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.588117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.588015 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0708855f-2a72-4f54-8ee5-2df4c073be69-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.589074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.588993 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:20.589324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.589249 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:20.589483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.589446 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "kube-api-access-cs72s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:20.589583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.589571 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo" pod "0708855f-2a72-4f54-8ee5-2df4c073be69" (UID: "0708855f-2a72-4f54-8ee5-2df4c073be69"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:20.689333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.689299 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs72s\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-kube-api-access-cs72s\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.689333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.689330 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.689333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.689340 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.690033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.689353 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0708855f-2a72-4f54-8ee5-2df4c073be69-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:20.699784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.699750 6171 generic.go:356] "Generic (PLEG): container finished" podID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerID="0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f" exitCode=0 Oct 30 06:20:20.699964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.699813 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" Oct 30 06:20:20.699964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.699843 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" event={"ID":"0708855f-2a72-4f54-8ee5-2df4c073be69","Type":"ContainerDied","Data":"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f"} Oct 30 06:20:20.699964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.699900 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj" event={"ID":"0708855f-2a72-4f54-8ee5-2df4c073be69","Type":"ContainerDied","Data":"2d1de47438aa9bc50a89af5fc2c469e3525f56e136328033b3b4b9af8413af38"} Oct 30 06:20:20.699964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.699928 6171 scope.go:122] "RemoveContainer" containerID="0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f" Oct 30 06:20:20.725378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.725355 6171 scope.go:122] "RemoveContainer" containerID="0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f" Oct 30 06:20:20.725705 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:20.725682 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f\": container with ID starting with 0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f not found: ID does not exist" containerID="0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f" Oct 30 06:20:20.725757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.725715 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f"} err="failed to get container status \"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f\": rpc error: code = NotFound desc = could not find container \"0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f\": container with ID starting with 0c290dc7545e9569d9620e135df4158388c38d2e2c5986e938aa5557fe43d68f not found: ID does not exist" Oct 30 06:20:20.775117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.775006 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:20:20.781987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:20.781955 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-6b5b1e-openshift-default-66fc85c695-99rtj"] Oct 30 06:20:21.474233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.474205 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:20:21.474424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.474400 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" podUID="69bddcaa-d7e8-478d-a87f-89998720be22" containerName="payload-processing" containerID="cri-o://0cbb7e27363d4e451924651baaf39a94ba6217a86948c57a7d2af385de05af61" gracePeriod=30 Oct 30 06:20:21.497494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.497393 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:20:21.497686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.497649 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" podUID="ea40ed47-98ce-47c7-b0d8-dacb65959077" containerName="payload-pre-processing" containerID="cri-o://8da316078834fc246001ac3a0c31e5105624f116aa6e0773fcc0eefe6a59e694" gracePeriod=30 Oct 30 06:20:21.689639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.688254 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" path="/var/lib/kubelet/pods/0708855f-2a72-4f54-8ee5-2df4c073be69/volumes" Oct 30 06:20:21.727973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.727880 6171 generic.go:356] "Generic (PLEG): container finished" podID="ea40ed47-98ce-47c7-b0d8-dacb65959077" containerID="8da316078834fc246001ac3a0c31e5105624f116aa6e0773fcc0eefe6a59e694" exitCode=0 Oct 30 06:20:21.728247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.728221 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" event={"ID":"ea40ed47-98ce-47c7-b0d8-dacb65959077","Type":"ContainerDied","Data":"8da316078834fc246001ac3a0c31e5105624f116aa6e0773fcc0eefe6a59e694"} Oct 30 06:20:21.735015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.731555 6171 generic.go:356] "Generic (PLEG): container finished" podID="69bddcaa-d7e8-478d-a87f-89998720be22" containerID="0cbb7e27363d4e451924651baaf39a94ba6217a86948c57a7d2af385de05af61" exitCode=0 Oct 30 06:20:21.735015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:21.731695 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" event={"ID":"69bddcaa-d7e8-478d-a87f-89998720be22","Type":"ContainerDied","Data":"0cbb7e27363d4e451924651baaf39a94ba6217a86948c57a7d2af385de05af61"} Oct 30 06:20:22.099130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.098289 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:20:22.134118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.134086 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:20:22.150190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.150132 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\" (UniqueName: \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\") pod \"69bddcaa-d7e8-478d-a87f-89998720be22\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " Oct 30 06:20:22.150370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.150317 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\") pod \"69bddcaa-d7e8-478d-a87f-89998720be22\" (UID: \"69bddcaa-d7e8-478d-a87f-89998720be22\") " Oct 30 06:20:22.150849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.150819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume" pod "69bddcaa-d7e8-478d-a87f-89998720be22" (UID: "69bddcaa-d7e8-478d-a87f-89998720be22"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:22.151203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.151179 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69bddcaa-d7e8-478d-a87f-89998720be22-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.152857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.152831 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x" pod "69bddcaa-d7e8-478d-a87f-89998720be22" (UID: "69bddcaa-d7e8-478d-a87f-89998720be22"). InnerVolumeSpecName "kube-api-access-9kd2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:22.228366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.228325 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf"] Oct 30 06:20:22.234517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.234380 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dddc53-tqvvf"] Oct 30 06:20:22.252092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.252055 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\" (UniqueName: \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\") pod \"ea40ed47-98ce-47c7-b0d8-dacb65959077\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " Oct 30 06:20:22.252254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.252240 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\") pod \"ea40ed47-98ce-47c7-b0d8-dacb65959077\" (UID: \"ea40ed47-98ce-47c7-b0d8-dacb65959077\") " Oct 30 06:20:22.252747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.252682 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume" pod "ea40ed47-98ce-47c7-b0d8-dacb65959077" (UID: "ea40ed47-98ce-47c7-b0d8-dacb65959077"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:22.252970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.252866 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kd2x\" (UniqueName: \"kubernetes.io/projected/69bddcaa-d7e8-478d-a87f-89998720be22-kube-api-access-9kd2x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.252970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.252881 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea40ed47-98ce-47c7-b0d8-dacb65959077-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.254418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.254389 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz" pod "ea40ed47-98ce-47c7-b0d8-dacb65959077" (UID: "ea40ed47-98ce-47c7-b0d8-dacb65959077"). InnerVolumeSpecName "kube-api-access-slbdz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:22.362512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.357799 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slbdz\" (UniqueName: \"kubernetes.io/projected/ea40ed47-98ce-47c7-b0d8-dacb65959077-kube-api-access-slbdz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:22.743318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.743290 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" Oct 30 06:20:22.743756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.743328 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg" event={"ID":"69bddcaa-d7e8-478d-a87f-89998720be22","Type":"ContainerDied","Data":"a63a8dcf299cf9ba60e9c72d81ab568a668c6cf5ea9668640e909bb2b985cc62"} Oct 30 06:20:22.743756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.743393 6171 scope.go:122] "RemoveContainer" containerID="0cbb7e27363d4e451924651baaf39a94ba6217a86948c57a7d2af385de05af61" Oct 30 06:20:22.745141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.745089 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" Oct 30 06:20:22.745359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.745083 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt" event={"ID":"ea40ed47-98ce-47c7-b0d8-dacb65959077","Type":"ContainerDied","Data":"52dcacbf773f530e527b1464d523124e42988e3471c2681acaab3e562ac0961f"} Oct 30 06:20:22.757874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.757849 6171 scope.go:122] "RemoveContainer" containerID="8da316078834fc246001ac3a0c31e5105624f116aa6e0773fcc0eefe6a59e694" Oct 30 06:20:22.804286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.804257 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:20:22.817987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.817963 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dddc53-69c8b8dc95-lwbgt"] Oct 30 06:20:22.846352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.846320 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:20:22.850996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:22.850974 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dddc53-9b584966d-lrxjg"] Oct 30 06:20:23.679811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:23.679751 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69bddcaa-d7e8-478d-a87f-89998720be22" path="/var/lib/kubelet/pods/69bddcaa-d7e8-478d-a87f-89998720be22/volumes" Oct 30 06:20:23.680473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:23.680448 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea40ed47-98ce-47c7-b0d8-dacb65959077" path="/var/lib/kubelet/pods/ea40ed47-98ce-47c7-b0d8-dacb65959077/volumes" Oct 30 06:20:23.681143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:23.681122 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb577654-0c8a-4647-a2ea-4e796c1bb89c" path="/var/lib/kubelet/pods/fb577654-0c8a-4647-a2ea-4e796c1bb89c/volumes" Oct 30 06:20:24.230325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.230279 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:20:24.232920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.232890 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b1155157-f3bc-40c7-b0fb-1922161e9196" containerName="revoke-keys" Oct 30 06:20:24.233119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.233101 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1155157-f3bc-40c7-b0fb-1922161e9196" containerName="revoke-keys" Oct 30 06:20:24.233337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.233319 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" Oct 30 06:20:24.233571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.233518 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" Oct 30 06:20:24.233747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.233730 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ea40ed47-98ce-47c7-b0d8-dacb65959077" containerName="payload-pre-processing" Oct 30 06:20:24.233918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.233900 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea40ed47-98ce-47c7-b0d8-dacb65959077" containerName="payload-pre-processing" Oct 30 06:20:24.234062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.234030 6171 cpu_manager.go:414] "containerMap: removing container" podUID="69bddcaa-d7e8-478d-a87f-89998720be22" containerName="payload-processing" Oct 30 06:20:24.234174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.234160 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69bddcaa-d7e8-478d-a87f-89998720be22" containerName="payload-processing" Oct 30 06:20:24.234661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.234644 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1155157-f3bc-40c7-b0fb-1922161e9196" containerName="revoke-keys" Oct 30 06:20:24.234805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.234793 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0708855f-2a72-4f54-8ee5-2df4c073be69" containerName="istio-proxy" Oct 30 06:20:24.234940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.234928 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea40ed47-98ce-47c7-b0d8-dacb65959077" containerName="payload-pre-processing" Oct 30 06:20:24.235037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.235027 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="69bddcaa-d7e8-478d-a87f-89998720be22" containerName="payload-processing" Oct 30 06:20:24.267426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.267392 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.315977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.315355 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d51cd8\"" Oct 30 06:20:24.315977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.315465 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d51cd8-dockercfg-kffjc\"" Oct 30 06:20:24.318397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.318345 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:20:24.391593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.391254 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.391593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.391392 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8cms\" (UniqueName: \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.493226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.492583 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.493226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.492750 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8cms\" (UniqueName: \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.493741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.493711 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.507713 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:24.507679 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefd2020_07b8_49de_9b70_61b8afc90036.slice/crio-3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfefd2020_07b8_49de_9b70_61b8afc90036.slice/crio-conmon-3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:20:24.531328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.523141 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8cms\" (UniqueName: \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\") pod \"payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.643783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.643755 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:24.731963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.731534 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:20:24.758707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.758683 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:20:24.784658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.783834 6171 generic.go:356] "Generic (PLEG): container finished" podID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerID="2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df" exitCode=0 Oct 30 06:20:24.784658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.784193 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" event={"ID":"4d67f174-f80a-452d-ae16-384c4f5ab28f","Type":"ContainerDied","Data":"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df"} Oct 30 06:20:24.784658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.784247 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" event={"ID":"4d67f174-f80a-452d-ae16-384c4f5ab28f","Type":"ContainerDied","Data":"5ef1712e80249d5c1fc568cfaa939837496680fbd9968dc795a66ac167a0518a"} Oct 30 06:20:24.784658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.784279 6171 scope.go:122] "RemoveContainer" containerID="2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df" Oct 30 06:20:24.784658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.784405 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn" Oct 30 06:20:24.789003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.788658 6171 generic.go:356] "Generic (PLEG): container finished" podID="fefd2020-07b8-49de-9b70-61b8afc90036" containerID="3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb" exitCode=0 Oct 30 06:20:24.789003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.788715 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" event={"ID":"fefd2020-07b8-49de-9b70-61b8afc90036","Type":"ContainerDied","Data":"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb"} Oct 30 06:20:24.789003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.788759 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" event={"ID":"fefd2020-07b8-49de-9b70-61b8afc90036","Type":"ContainerDied","Data":"3f7d4b98aa5c4ecad20dc1e4b7ef24b8fde0dd6c0c5dd595a0278761c84a87db"} Oct 30 06:20:24.789003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.788824 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc" Oct 30 06:20:24.804234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.804206 6171 scope.go:122] "RemoveContainer" containerID="2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df" Oct 30 06:20:24.804563 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:24.804544 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df\": container with ID starting with 2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df not found: ID does not exist" containerID="2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df" Oct 30 06:20:24.804630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.804571 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df"} err="failed to get container status \"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df\": rpc error: code = NotFound desc = could not find container \"2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df\": container with ID starting with 2217a84327effca7b119806a96ce1f27f5ad8b4659033222531681c6c2a6c8df not found: ID does not exist" Oct 30 06:20:24.804630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.804588 6171 scope.go:122] "RemoveContainer" containerID="3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb" Oct 30 06:20:24.811474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.811439 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\") pod \"4d67f174-f80a-452d-ae16-384c4f5ab28f\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " Oct 30 06:20:24.811625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.811592 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\") pod \"fefd2020-07b8-49de-9b70-61b8afc90036\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " Oct 30 06:20:24.811702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.811632 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\" (UniqueName: \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\") pod \"4d67f174-f80a-452d-ae16-384c4f5ab28f\" (UID: \"4d67f174-f80a-452d-ae16-384c4f5ab28f\") " Oct 30 06:20:24.812146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.812116 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume" pod "4d67f174-f80a-452d-ae16-384c4f5ab28f" (UID: "4d67f174-f80a-452d-ae16-384c4f5ab28f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:24.812438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.812408 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d67f174-f80a-452d-ae16-384c4f5ab28f-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:24.812565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.812537 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume" pod "fefd2020-07b8-49de-9b70-61b8afc90036" (UID: "fefd2020-07b8-49de-9b70-61b8afc90036"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:24.815093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.815009 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z" pod "4d67f174-f80a-452d-ae16-384c4f5ab28f" (UID: "4d67f174-f80a-452d-ae16-384c4f5ab28f"). InnerVolumeSpecName "kube-api-access-t2f2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:24.822403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.822106 6171 scope.go:122] "RemoveContainer" containerID="3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb" Oct 30 06:20:24.822528 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:24.822501 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb\": container with ID starting with 3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb not found: ID does not exist" containerID="3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb" Oct 30 06:20:24.822589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.822543 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb"} err="failed to get container status \"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb\": rpc error: code = NotFound desc = could not find container \"3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb\": container with ID starting with 3bdd4d8e3f799ed5ec608a5e2fec56f59461368dd1d6c5510b0fd69ea82786cb not found: ID does not exist" Oct 30 06:20:24.913207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.913161 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\" (UniqueName: \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\") pod \"fefd2020-07b8-49de-9b70-61b8afc90036\" (UID: \"fefd2020-07b8-49de-9b70-61b8afc90036\") " Oct 30 06:20:24.913846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.913826 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fefd2020-07b8-49de-9b70-61b8afc90036-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:24.913909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.913850 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2f2z\" (UniqueName: \"kubernetes.io/projected/4d67f174-f80a-452d-ae16-384c4f5ab28f-kube-api-access-t2f2z\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:24.915552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:24.915533 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq" pod "fefd2020-07b8-49de-9b70-61b8afc90036" (UID: "fefd2020-07b8-49de-9b70-61b8afc90036"). InnerVolumeSpecName "kube-api-access-f6fzq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:25.001007 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:25.000970 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode93b6a22_40a9_4509_ae84_c77201323642.slice/crio-01fee9d9f4ef24cfeb1ce09b8606a82f842e0b3042f4923978751dc54d24bf0d WatchSource:0}: Error finding container 01fee9d9f4ef24cfeb1ce09b8606a82f842e0b3042f4923978751dc54d24bf0d: Status 404 returned error can't find the container with id 01fee9d9f4ef24cfeb1ce09b8606a82f842e0b3042f4923978751dc54d24bf0d Oct 30 06:20:25.002079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.002034 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:20:25.015516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.015479 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6fzq\" (UniqueName: \"kubernetes.io/projected/fefd2020-07b8-49de-9b70-61b8afc90036-kube-api-access-f6fzq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:25.176986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.176858 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:20:25.183304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.183266 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-d1bfce-5c48ff6b99-cfxfn"] Oct 30 06:20:25.218759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.218724 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:20:25.236631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.236573 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w5-d1bfce-574bdf5d65-nwhhc"] Oct 30 06:20:25.683877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.683786 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" path="/var/lib/kubelet/pods/4d67f174-f80a-452d-ae16-384c4f5ab28f/volumes" Oct 30 06:20:25.684343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.684320 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" path="/var/lib/kubelet/pods/fefd2020-07b8-49de-9b70-61b8afc90036/volumes" Oct 30 06:20:25.804052 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.804006 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" event={"ID":"e93b6a22-40a9-4509-ae84-c77201323642","Type":"ContainerStarted","Data":"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062"} Oct 30 06:20:25.804266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.804096 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:25.804266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.804119 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" event={"ID":"e93b6a22-40a9-4509-ae84-c77201323642","Type":"ContainerStarted","Data":"01fee9d9f4ef24cfeb1ce09b8606a82f842e0b3042f4923978751dc54d24bf0d"} Oct 30 06:20:25.849350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:25.849300 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" podStartSLOduration=1.8492871819999999 podStartE2EDuration="1.849287182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:25.848868764 +0000 UTC m=+2884.289245957" watchObservedRunningTime="2026-10-30 06:20:25.849287182 +0000 UTC m=+2884.289664375" Oct 30 06:20:26.384392 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:26.384355 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:20:29.345735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.345696 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:20:29.346219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.345922 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" containerID="cri-o://91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433" gracePeriod=30 Oct 30 06:20:29.458873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.457568 6171 scope.go:122] "RemoveContainer" containerID="9dd1fc4555fd600b013e6e5549a29f7dc5b65ca423e59877907443fb309bc4f7" Oct 30 06:20:29.577305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.571822 6171 scope.go:122] "RemoveContainer" containerID="4ca204e414285ee69cc6842b3f9efcdcabcbec4daf523ec1ec325ff48234d4e2" Oct 30 06:20:29.643074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.639551 6171 scope.go:122] "RemoveContainer" containerID="1c7ca181e05f3cd69c93d805593b97a33f18d90d8d926f1d2e256c9a98b6477b" Oct 30 06:20:29.719907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.719887 6171 scope.go:122] "RemoveContainer" containerID="7edcb5ffe6a208e01812c1b796d7294ba438115ebb73be8d009ac446bef4de2f" Oct 30 06:20:29.733834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.733804 6171 scope.go:122] "RemoveContainer" containerID="348ddc15e5a4d5905f97bc3abaf3c5e633afc38842195bf934c40741326153ac" Oct 30 06:20:29.744557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.744530 6171 scope.go:122] "RemoveContainer" containerID="934168f36c9cbb350ea1493da5617a9fdbdaebc7478037a66b2654e30b02609e" Oct 30 06:20:29.897896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.897802 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc"] Oct 30 06:20:29.898836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.898813 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" Oct 30 06:20:29.898966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.898837 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" Oct 30 06:20:29.898966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.898885 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" Oct 30 06:20:29.898966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.898893 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" Oct 30 06:20:29.899194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.899150 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d67f174-f80a-452d-ae16-384c4f5ab28f" containerName="payload-pre-processing" Oct 30 06:20:29.899194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.899176 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fefd2020-07b8-49de-9b70-61b8afc90036" containerName="payload-processing" Oct 30 06:20:29.905577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.905551 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:29.921542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:29.921517 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc"] Oct 30 06:20:30.000762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.000723 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.000977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.000949 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.001126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.001108 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xsxr\" (UniqueName: \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.009113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.009089 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:30.009256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.009224 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:20:30.102627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.102584 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xsxr\" (UniqueName: \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.102837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.102734 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.102932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.102896 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.103385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.103366 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.105644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.105598 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.128306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.128280 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xsxr\" (UniqueName: \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\") pod \"maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.222655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.222630 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:30.541611 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:30.541576 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fddd0e8_35bb_4b15_a6cf_6f91e5f0bd34.slice/crio-bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5 WatchSource:0}: Error finding container bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5: Status 404 returned error can't find the container with id bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5 Oct 30 06:20:30.542854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.542829 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc"] Oct 30 06:20:30.602356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.602330 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:20:30.608705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.608680 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.657450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.657402 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:20:30.664660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.664592 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:20:30.671380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.671351 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.672488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.672188 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:20:30.710644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710612 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.710813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710659 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.710813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710734 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcrmx\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.710895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710823 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.710937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710888 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.710937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710926 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710960 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.710990 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711183 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711286 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711322 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711355 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711532 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711582 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7ml7\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711631 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.711691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711683 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711793 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.711870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.711837 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832353 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832414 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832458 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832494 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832547 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832585 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832657 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832689 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832729 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832854 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832903 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7ml7\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832938 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.832986 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.833037 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.833132 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.833275 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.833327 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.833377 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcrmx\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.834219 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.834518 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.839123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.835099 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.840693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.839746 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.840693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.839958 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.840693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.840184 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.840880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.840724 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.842168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.840979 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.842168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.841623 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.847756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.847693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.848144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.848119 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.848561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.848532 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.850020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.849950 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.850846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.850795 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.861541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.861511 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcrmx\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.864986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.864958 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.879210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.879178 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7ml7\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:30.879426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.879403 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\") pod \"e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:30.894894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.894864 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" event={"ID":"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34","Type":"ContainerStarted","Data":"bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5"} Oct 30 06:20:30.925268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:30.925231 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:31.010572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.010541 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:31.259283 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:31.259256 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd11f58b_7756_452e_be90_d1e8b2b3dafd.slice/crio-42b064007b16f2e58f60ba785ca29adec3351cab4459a03812eff2bfff131df1 WatchSource:0}: Error finding container 42b064007b16f2e58f60ba785ca29adec3351cab4459a03812eff2bfff131df1: Status 404 returned error can't find the container with id 42b064007b16f2e58f60ba785ca29adec3351cab4459a03812eff2bfff131df1 Oct 30 06:20:31.260427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.260405 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:20:31.262720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.262689 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.262824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.262753 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.262824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.262784 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.396955 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:31.396926 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod172e7059_b3b9_4a9c_9441_1f2905b4ec3d.slice/crio-061d4fa5c9335a8bfd7a3f6fe2a0dacd0c7ecf14e11ce577862cc4ae41e01cf5 WatchSource:0}: Error finding container 061d4fa5c9335a8bfd7a3f6fe2a0dacd0c7ecf14e11ce577862cc4ae41e01cf5: Status 404 returned error can't find the container with id 061d4fa5c9335a8bfd7a3f6fe2a0dacd0c7ecf14e11ce577862cc4ae41e01cf5 Oct 30 06:20:31.398128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.398102 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:20:31.400948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.400908 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.401096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.400985 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.401096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.401016 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:31.914904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.914848 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" event={"ID":"fd11f58b-7756-452e-be90-d1e8b2b3dafd","Type":"ContainerStarted","Data":"9bd63cfed53ee35b54a37c48bbbc2bd9e5ce5363b699b1839aceec09aaacc010"} Oct 30 06:20:31.915578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.914920 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" event={"ID":"fd11f58b-7756-452e-be90-d1e8b2b3dafd","Type":"ContainerStarted","Data":"42b064007b16f2e58f60ba785ca29adec3351cab4459a03812eff2bfff131df1"} Oct 30 06:20:31.917653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.917623 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" event={"ID":"172e7059-b3b9-4a9c-9441-1f2905b4ec3d","Type":"ContainerStarted","Data":"8fa59c1d018244ae21cc2e4bcc6a804f8effd34700ae432b3f2ca965b808e37d"} Oct 30 06:20:31.917804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.917687 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" event={"ID":"172e7059-b3b9-4a9c-9441-1f2905b4ec3d","Type":"ContainerStarted","Data":"061d4fa5c9335a8bfd7a3f6fe2a0dacd0c7ecf14e11ce577862cc4ae41e01cf5"} Oct 30 06:20:31.920468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.920441 6171 generic.go:356] "Generic (PLEG): container finished" podID="8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" containerID="afad4057133708057278d787949e3f87c046346ccd789334cf66e10589271d72" exitCode=0 Oct 30 06:20:31.920602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.920511 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" event={"ID":"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34","Type":"ContainerDied","Data":"afad4057133708057278d787949e3f87c046346ccd789334cf66e10589271d72"} Oct 30 06:20:31.925639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.925612 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:31.968689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:31.968590 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podStartSLOduration=1.968575638 podStartE2EDuration="1.968575638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:31.965857779 +0000 UTC m=+2890.406234971" watchObservedRunningTime="2026-10-30 06:20:31.968575638 +0000 UTC m=+2890.408952830" Oct 30 06:20:32.011381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:32.011338 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:32.013219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:32.013194 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:32.013331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:32.013244 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:32.115137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:32.114393 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podStartSLOduration=2.114371958 podStartE2EDuration="2.114371958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:32.111438766 +0000 UTC m=+2890.551815959" watchObservedRunningTime="2026-10-30 06:20:32.114371958 +0000 UTC m=+2890.554749154" Oct 30 06:20:32.990089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:32.981338 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:20:33.130077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.079365 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:33.130077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.079430 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:33.130077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.080550 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:33.130077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.080602 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:33.174861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.155295 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:20:33.174861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.155922 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.208008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.207974 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5deb36\"" Oct 30 06:20:33.278970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.278925 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sm8p\" (UniqueName: \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.279291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.279269 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.417070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.405359 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.417070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.405607 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sm8p\" (UniqueName: \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.438174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.437766 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.444112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.441584 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sm8p\" (UniqueName: \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\") pod \"payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.553665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.553641 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:33.772495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.772458 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:33.832070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.809870 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:20:33.832070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.811310 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" containerName="revoke-keys" Oct 30 06:20:33.832070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.811332 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" containerName="revoke-keys" Oct 30 06:20:33.832070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.811632 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" containerName="revoke-keys" Oct 30 06:20:33.854161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.854131 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:20:33.854161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.854166 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:20:33.855267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.855219 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.860519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.860497 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:20:33.860782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.860762 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.873527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.871974 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\") pod \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " Oct 30 06:20:33.873527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.872315 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\") pod \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " Oct 30 06:20:33.873527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.872556 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\" (UniqueName: \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\") pod \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\" (UID: \"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34\") " Oct 30 06:20:33.873527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.873030 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca" pod "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" (UID: "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:33.873527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.873403 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:33.876273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.876250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca" pod "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" (UID: "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:33.938151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.936514 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:33.938151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.936572 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:33.948689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.948658 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr" pod "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" (UID: "8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34"). InnerVolumeSpecName "kube-api-access-9xsxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:33.975392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975361 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.975502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975408 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.975591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975564 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975663 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975746 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.975817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975803 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975831 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975866 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975892 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.975970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975926 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmlwt\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.976005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.975976 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.976151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.976131 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.976248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.976194 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.976309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.976275 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:33.976309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.976302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.976462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.976444 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.977220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.977195 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.977299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.977239 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqf9\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:33.977468 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.977451 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xsxr\" (UniqueName: \"kubernetes.io/projected/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-kube-api-access-9xsxr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:33.977517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:33.977475 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:34.041074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.040386 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:34.041074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.040446 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:34.079068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.078977 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079033 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079178 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079197 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079219 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079248 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079272 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmlwt\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079338 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079352 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079401 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079478 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079498 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079554 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079630 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.079701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079651 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqf9\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.080330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079743 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.080330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.079758 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.080330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.080315 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.096066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.089090 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.121069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.114423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.121069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.115188 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.121069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.116388 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.137462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.137422 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.137637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.137544 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.137716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.137692 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.137776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.137751 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.140893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.139700 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.140893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.140452 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.142287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.142256 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmlwt\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.142601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.142573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.142716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.142693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.143218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.143179 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.143846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.143621 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqf9\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.143846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.143812 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\") pod \"e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.233682 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:34.233626 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4a02500_dd1d_49cd_a510_6b93254bafac.slice/crio-2dc3781605ff0a546c1aa86665d4cd27e4d1a1cf75594ac59b79fd7c92f69982 WatchSource:0}: Error finding container 2dc3781605ff0a546c1aa86665d4cd27e4d1a1cf75594ac59b79fd7c92f69982: Status 404 returned error can't find the container with id 2dc3781605ff0a546c1aa86665d4cd27e4d1a1cf75594ac59b79fd7c92f69982 Oct 30 06:20:34.233901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.233853 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:34.240923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.240898 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:20:34.250424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.250352 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:34.264372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.264210 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" event={"ID":"e4a02500-dd1d-49cd-a510-6b93254bafac","Type":"ContainerStarted","Data":"2dc3781605ff0a546c1aa86665d4cd27e4d1a1cf75594ac59b79fd7c92f69982"} Oct 30 06:20:34.268819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.268764 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" Oct 30 06:20:34.268922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.268873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc" event={"ID":"8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34","Type":"ContainerDied","Data":"bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5"} Oct 30 06:20:34.269004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.268923 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcfb1a7099146b48db8d8bdc49a373a4a956e34dc0ef1fb786fcba1dcc10d7a5" Oct 30 06:20:34.943577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.943492 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:34.943818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.943790 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:34.971608 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:34.971575 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d136afb_70b6_4dd4_9baf_030e53aacb49.slice/crio-3306be7e7823d042612b3dadef790391fbd4d4e2de7e6275def82acffd433ceb WatchSource:0}: Error finding container 3306be7e7823d042612b3dadef790391fbd4d4e2de7e6275def82acffd433ceb: Status 404 returned error can't find the container with id 3306be7e7823d042612b3dadef790391fbd4d4e2de7e6275def82acffd433ceb Oct 30 06:20:34.976311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.976267 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.976548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.976523 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.976682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.976665 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:34.977201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:34.977174 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:20:35.059235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.057391 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:35.059235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.057481 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:35.059235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.057526 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:20:35.060658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.060232 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:35.060658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.060307 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:35.075219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.075187 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:20:35.173544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.173519 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:20:35.263367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263337 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263413 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263465 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263521 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263626 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263735 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.263927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.263913 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.264032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.264019 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.264100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.264069 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\") pod \"7301aaef-11d6-4579-9a2b-a230ea5665f1\" (UID: \"7301aaef-11d6-4579-9a2b-a230ea5665f1\") " Oct 30 06:20:35.265780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.265581 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:35.266124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.265905 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:35.266124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.265932 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:35.266286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.266235 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:35.266506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.266481 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:35.269560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.269507 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:35.270130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.270102 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:35.270304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.270281 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:35.271273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.271253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z" pod "7301aaef-11d6-4579-9a2b-a230ea5665f1" (UID: "7301aaef-11d6-4579-9a2b-a230ea5665f1"). InnerVolumeSpecName "kube-api-access-kls4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:35.368127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.364995 6171 generic.go:356] "Generic (PLEG): container finished" podID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerID="91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433" exitCode=0 Oct 30 06:20:35.368127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.365087 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerDied","Data":"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433"} Oct 30 06:20:35.368127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.365128 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" event={"ID":"7301aaef-11d6-4579-9a2b-a230ea5665f1","Type":"ContainerDied","Data":"34486dc0564ddd4ad68893fce0911d258cad1728ed7be8f139e28b8852e6e2c7"} Oct 30 06:20:35.368127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.365152 6171 scope.go:122] "RemoveContainer" containerID="91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433" Oct 30 06:20:35.368127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.365313 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369842 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kls4z\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-kube-api-access-kls4z\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369873 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369889 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369915 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7301aaef-11d6-4579-9a2b-a230ea5665f1-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369931 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369945 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369958 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369971 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7301aaef-11d6-4579-9a2b-a230ea5665f1-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.375768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.369998 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7301aaef-11d6-4579-9a2b-a230ea5665f1-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:35.386014 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.383029 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" event={"ID":"fee094ed-58ff-4d39-99ff-8d7c9eb900b9","Type":"ContainerStarted","Data":"d1c19390c7b1f23aff93fb4da6e9db4d383ceb3e0b0ae2f4f7b41b2a8f4854f3"} Oct 30 06:20:35.389698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.387864 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" event={"ID":"0d136afb-70b6-4dd4-9baf-030e53aacb49","Type":"ContainerStarted","Data":"3306be7e7823d042612b3dadef790391fbd4d4e2de7e6275def82acffd433ceb"} Oct 30 06:20:35.392547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.391349 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" event={"ID":"e4a02500-dd1d-49cd-a510-6b93254bafac","Type":"ContainerStarted","Data":"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7"} Oct 30 06:20:35.392547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.392481 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:35.464211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.463485 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" podStartSLOduration=3.46346386 podStartE2EDuration="3.46346386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:35.449519541 +0000 UTC m=+2893.889896736" watchObservedRunningTime="2026-10-30 06:20:35.46346386 +0000 UTC m=+2893.903841056" Oct 30 06:20:35.866318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.866280 6171 scope.go:122] "RemoveContainer" containerID="8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa" Oct 30 06:20:35.939071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.931596 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:35.939071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:35.931679 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:36.048947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.025299 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:36.063299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.061682 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:36.301584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.301552 6171 scope.go:122] "RemoveContainer" containerID="91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433" Oct 30 06:20:36.302383 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:36.302333 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433\": container with ID starting with 91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433 not found: ID does not exist" containerID="91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433" Oct 30 06:20:36.302572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.302543 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433"} err="failed to get container status \"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433\": rpc error: code = NotFound desc = could not find container \"91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433\": container with ID starting with 91abd4e3033633006604ead7dc74f116d4272221f1707780cf10020151b5d433 not found: ID does not exist" Oct 30 06:20:36.302723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.302709 6171 scope.go:122] "RemoveContainer" containerID="8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa" Oct 30 06:20:36.303329 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:36.303304 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa\": container with ID starting with 8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa not found: ID does not exist" containerID="8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa" Oct 30 06:20:36.303565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.303541 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa"} err="failed to get container status \"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa\": rpc error: code = NotFound desc = could not find container \"8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa\": container with ID starting with 8f65ae0d3073e3223a8229273b4c07a9c21b8a53112845ccca93ffce6b6ea1aa not found: ID does not exist" Oct 30 06:20:36.603127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.601725 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:20:36.671993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.662666 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" event={"ID":"fee094ed-58ff-4d39-99ff-8d7c9eb900b9","Type":"ContainerStarted","Data":"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71"} Oct 30 06:20:36.769235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.756670 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" event={"ID":"0d136afb-70b6-4dd4-9baf-030e53aacb49","Type":"ContainerStarted","Data":"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf"} Oct 30 06:20:36.813071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.801150 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dddc53-openshift-default-69ddd67fc-tgqf6"] Oct 30 06:20:36.862321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.862234 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:20:36.903208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.903133 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw"] Oct 30 06:20:36.947071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.934250 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:36.947071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:36.934325 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:37.097148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.089316 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-03d349-k5nbw"] Oct 30 06:20:37.211130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.186815 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podStartSLOduration=4.186791818 podStartE2EDuration="4.186791818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:36.763755805 +0000 UTC m=+2895.204133000" watchObservedRunningTime="2026-10-30 06:20:37.186791818 +0000 UTC m=+2895.627169013" Oct 30 06:20:37.398864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.398825 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.461080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.437256 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.461080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.437322 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:37.461080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.438648 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:37.461080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.399152 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:37.550455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.523909 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:37.551353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.551322 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.551623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.551568 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:37.925126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.924210 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cc76f55-5777-4bb7-9e91-b2c4469775a0" path="/var/lib/kubelet/pods/4cc76f55-5777-4bb7-9e91-b2c4469775a0/volumes" Oct 30 06:20:37.935071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.931476 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" path="/var/lib/kubelet/pods/7301aaef-11d6-4579-9a2b-a230ea5665f1/volumes" Oct 30 06:20:37.974073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.973319 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" start-of-body= Oct 30 06:20:37.974073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.973384 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.136:15021/healthz/ready\": dial tcp 10.133.1.136:15021: connect: connection refused" Oct 30 06:20:37.994107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:37.991942 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podStartSLOduration=4.991918542 podStartE2EDuration="4.991918542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:36.83569035 +0000 UTC m=+2895.276067547" watchObservedRunningTime="2026-10-30 06:20:37.991918542 +0000 UTC m=+2896.432295737" Oct 30 06:20:38.015759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.015722 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:38.019172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.019133 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:38.232137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.224141 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:20:38.268071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.247785 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.268071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.253379 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:38.268071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.253432 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:38.268071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.253920 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:38.268071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.253950 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:38.434071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.430359 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.470070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.469432 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.499128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.498356 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.531190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.531147 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.661554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.661508 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.967136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.939410 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:38.986232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:38.986187 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.114:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:39.012894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.012859 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" start-of-body= Oct 30 06:20:39.013205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.013173 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.137:15021/healthz/ready\": dial tcp 10.133.1.137:15021: connect: connection refused" Oct 30 06:20:39.249483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.249354 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" probeResult="failure" output="Get \"https://10.133.0.124:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:39.267611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.267565 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" probeResult="failure" output="Get \"https://10.133.0.62:8000/ready\": context deadline exceeded" Oct 30 06:20:39.269751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.269722 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:20:39.269895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.269786 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:20:39.354117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.351408 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:39.354117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.351467 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:39.373435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.373378 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:39.425098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.412444 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:39.441192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.441150 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": context deadline exceeded" Oct 30 06:20:39.441574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.441556 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:20:39.510151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.502213 6171 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/health\": context deadline exceeded" Oct 30 06:20:39.560085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.538275 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" probeResult="failure" output="Get \"https://10.133.0.126:8000/ready\": context deadline exceeded" Oct 30 06:20:39.560085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.538728 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:20:39.693106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.678365 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:39.693106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.679840 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:39.986129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.974678 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:39.992154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:39.986808 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" probeResult="failure" output="Get \"https://10.133.1.114:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:40.070070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.067856 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:20:40.070070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.068920 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:20:40.092075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.073459 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:20:40.092075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.075795 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:20:40.092075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.077594 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:20:40.092075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.077808 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:20:40.092075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.078179 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:20:40.161996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.123159 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:40.161996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.125833 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:40.161996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.129931 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:20:40.161996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.149353 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:20:40.161996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.157092 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:20:40.248469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.248331 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:40.248806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.248757 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:40.256417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.254645 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:40.256417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.254699 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:40.859087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:40.853595 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:20:41.063274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:41.063242 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:20:41.242065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:41.240280 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:41.242065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:41.240351 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:41.252869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:41.252827 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:41.253070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:41.252989 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:42.243088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:42.241569 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:42.243088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:42.241634 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:42.253111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:42.253057 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" start-of-body= Oct 30 06:20:42.253370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:42.253121 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.140:15021/healthz/ready\": dial tcp 10.133.1.140:15021: connect: connection refused" Oct 30 06:20:43.235102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:43.235023 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" start-of-body= Oct 30 06:20:43.235302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:43.235131 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.139:15021/healthz/ready\": dial tcp 10.133.1.139:15021: connect: connection refused" Oct 30 06:20:43.303981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:43.303931 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:43.305022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:43.304995 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:43.305819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:43.305790 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:20:44.239831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.239799 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:44.240574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.240340 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:44.241120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.241100 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:20:44.644188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.636418 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:20:44.644188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.636690 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" podUID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" containerName="payload-processing" containerID="cri-o://e98d0d9acb5c9330f4908aba45201cf359615565c754d922bc3399faca9c72ec" gracePeriod=30 Oct 30 06:20:44.702377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.694121 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:20:44.702377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.694336 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" podUID="8aaf8aff-680a-4c29-a191-854acf6204a0" containerName="payload-pre-processing" containerID="cri-o://ea5283849e8a98a3c6102a5158678247ba65612a08013f895980974101f7c20e" gracePeriod=30 Oct 30 06:20:44.906217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.906182 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk"] Oct 30 06:20:44.909379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.907624 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:44.909379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.907650 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:44.909379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.907914 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:44.909379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.907945 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:44.917129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.916193 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:44.946224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.942818 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:44.946224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.943600 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:20:45.017402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.999740 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwhtq\" (UniqueName: \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.017402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:44.999905 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.017402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.000118 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.065407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.064472 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:20:45.065407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.064720 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerName="maas-api" containerID="cri-o://4633f60de251c6fcb5c5dec4342b7d030b096aadbc8e03430bc7be2498085e6e" gracePeriod=30 Oct 30 06:20:45.132309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.131825 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.155602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.155350 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwhtq\" (UniqueName: \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.155602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.155518 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.156593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.152225 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.249070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.248987 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.339272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.272927 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwhtq\" (UniqueName: \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\") pod \"maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.384089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.379505 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk"] Oct 30 06:20:45.401174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.401145 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:45.447829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.447798 6171 generic.go:356] "Generic (PLEG): container finished" podID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerID="4633f60de251c6fcb5c5dec4342b7d030b096aadbc8e03430bc7be2498085e6e" exitCode=0 Oct 30 06:20:45.457129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.448013 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" event={"ID":"d6594fe0-7c2f-4344-9167-0d507b0232ce","Type":"ContainerDied","Data":"4633f60de251c6fcb5c5dec4342b7d030b096aadbc8e03430bc7be2498085e6e"} Oct 30 06:20:45.466859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.466829 6171 generic.go:356] "Generic (PLEG): container finished" podID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" containerID="e98d0d9acb5c9330f4908aba45201cf359615565c754d922bc3399faca9c72ec" exitCode=0 Oct 30 06:20:45.467173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.466910 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" event={"ID":"3a25279c-bb5e-47fb-9b38-842b88ac29a4","Type":"ContainerDied","Data":"e98d0d9acb5c9330f4908aba45201cf359615565c754d922bc3399faca9c72ec"} Oct 30 06:20:45.468083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.468012 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:20:45.470850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.470261 6171 generic.go:356] "Generic (PLEG): container finished" podID="8aaf8aff-680a-4c29-a191-854acf6204a0" containerID="ea5283849e8a98a3c6102a5158678247ba65612a08013f895980974101f7c20e" exitCode=0 Oct 30 06:20:45.473919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.470324 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" event={"ID":"8aaf8aff-680a-4c29-a191-854acf6204a0","Type":"ContainerDied","Data":"ea5283849e8a98a3c6102a5158678247ba65612a08013f895980974101f7c20e"} Oct 30 06:20:45.692099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.692000 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\") pod \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " Oct 30 06:20:45.692569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.692345 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\" (UniqueName: \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\") pod \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\" (UID: \"3a25279c-bb5e-47fb-9b38-842b88ac29a4\") " Oct 30 06:20:45.693897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.693872 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume" pod "3a25279c-bb5e-47fb-9b38-842b88ac29a4" (UID: "3a25279c-bb5e-47fb-9b38-842b88ac29a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:45.703295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.700387 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.45:8443/health\": dial tcp 10.133.1.45:8443: connect: connection refused" Oct 30 06:20:45.703295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.701209 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9" pod "3a25279c-bb5e-47fb-9b38-842b88ac29a4" (UID: "3a25279c-bb5e-47fb-9b38-842b88ac29a4"). InnerVolumeSpecName "kube-api-access-fc4p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:45.776872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.776555 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:20:45.798728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.798546 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc"] Oct 30 06:20:45.799232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.799205 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc4p9\" (UniqueName: \"kubernetes.io/projected/3a25279c-bb5e-47fb-9b38-842b88ac29a4-kube-api-access-fc4p9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.799926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.799905 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a25279c-bb5e-47fb-9b38-842b88ac29a4-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:45.805467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.805441 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-f7ad21-kxmfc"] Oct 30 06:20:45.946124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.945368 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\" (UniqueName: \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\") pod \"8aaf8aff-680a-4c29-a191-854acf6204a0\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " Oct 30 06:20:45.946124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.945475 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\") pod \"8aaf8aff-680a-4c29-a191-854acf6204a0\" (UID: \"8aaf8aff-680a-4c29-a191-854acf6204a0\") " Oct 30 06:20:45.961286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:45.950259 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume" pod "8aaf8aff-680a-4c29-a191-854acf6204a0" (UID: "8aaf8aff-680a-4c29-a191-854acf6204a0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:46.065583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.057174 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8aaf8aff-680a-4c29-a191-854acf6204a0-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.082820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.081250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57" pod "8aaf8aff-680a-4c29-a191-854acf6204a0" (UID: "8aaf8aff-680a-4c29-a191-854acf6204a0"). InnerVolumeSpecName "kube-api-access-c5j57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:46.160873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.160841 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5j57\" (UniqueName: \"kubernetes.io/projected/8aaf8aff-680a-4c29-a191-854acf6204a0-kube-api-access-c5j57\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.193891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.193854 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:20:46.276115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.276083 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk"] Oct 30 06:20:46.326391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.326360 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw"] Oct 30 06:20:46.331541 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.331518 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-d1bfce-4nfkw"] Oct 30 06:20:46.363755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.363726 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\" (UniqueName: \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\") pod \"d6594fe0-7c2f-4344-9167-0d507b0232ce\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " Oct 30 06:20:46.363908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.363890 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") pod \"d6594fe0-7c2f-4344-9167-0d507b0232ce\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " Oct 30 06:20:46.364177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.364154 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\") pod \"d6594fe0-7c2f-4344-9167-0d507b0232ce\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " Oct 30 06:20:46.364294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.364278 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\") pod \"d6594fe0-7c2f-4344-9167-0d507b0232ce\" (UID: \"d6594fe0-7c2f-4344-9167-0d507b0232ce\") " Oct 30 06:20:46.364547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.364502 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca" pod "d6594fe0-7c2f-4344-9167-0d507b0232ce" (UID: "d6594fe0-7c2f-4344-9167-0d507b0232ce"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:46.365182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.365118 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d6594fe0-7c2f-4344-9167-0d507b0232ce-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.366354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.366324 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw" pod "d6594fe0-7c2f-4344-9167-0d507b0232ce" (UID: "d6594fe0-7c2f-4344-9167-0d507b0232ce"). InnerVolumeSpecName "kube-api-access-9k7sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:46.366759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.366740 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs" pod "d6594fe0-7c2f-4344-9167-0d507b0232ce" (UID: "d6594fe0-7c2f-4344-9167-0d507b0232ce"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:46.366862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.366832 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls" pod "d6594fe0-7c2f-4344-9167-0d507b0232ce" (UID: "d6594fe0-7c2f-4344-9167-0d507b0232ce"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:46.466441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.466406 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.466441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.466434 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k7sw\" (UniqueName: \"kubernetes.io/projected/d6594fe0-7c2f-4344-9167-0d507b0232ce-kube-api-access-9k7sw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.466441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.466443 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d6594fe0-7c2f-4344-9167-0d507b0232ce-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:46.481180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.481144 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" event={"ID":"3a25279c-bb5e-47fb-9b38-842b88ac29a4","Type":"ContainerDied","Data":"bd8f66eb1d6ac87b3ef981e3003a993cf381f225d81e24475b34d83590ece827"} Oct 30 06:20:46.481330 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.481213 6171 scope.go:122] "RemoveContainer" containerID="e98d0d9acb5c9330f4908aba45201cf359615565c754d922bc3399faca9c72ec" Oct 30 06:20:46.481387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.481366 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp" Oct 30 06:20:46.484243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.484223 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" Oct 30 06:20:46.484498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.484250 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6" event={"ID":"8aaf8aff-680a-4c29-a191-854acf6204a0","Type":"ContainerDied","Data":"2a669fbe25f0f90d047b4bb3a4233d921a83c36184f8a0d25b27ff9a9a68c7c9"} Oct 30 06:20:46.486776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.486747 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" event={"ID":"d6594fe0-7c2f-4344-9167-0d507b0232ce","Type":"ContainerDied","Data":"b6bea80b939c5d031dfa4140373f93393bcf2b0b7b9c0f2677ffb4f88530455a"} Oct 30 06:20:46.486875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.486824 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf" Oct 30 06:20:46.488896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.488870 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" event={"ID":"e8722a33-9b7a-439f-9693-a05cea761700","Type":"ContainerStarted","Data":"ae07ace37085978790ed9821e07b52d3c6e834cec7215cca0796a8ccfb36c641"} Oct 30 06:20:46.508711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.508603 6171 scope.go:122] "RemoveContainer" containerID="ea5283849e8a98a3c6102a5158678247ba65612a08013f895980974101f7c20e" Oct 30 06:20:46.523395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.523372 6171 scope.go:122] "RemoveContainer" containerID="4633f60de251c6fcb5c5dec4342b7d030b096aadbc8e03430bc7be2498085e6e" Oct 30 06:20:46.550056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.550014 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:20:46.557237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.557215 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-d1bfce-64c44989ff-n5pzf"] Oct 30 06:20:46.608944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.608916 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:20:46.614060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.614014 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-f7ad21-c576654b6-5msgp"] Oct 30 06:20:46.650460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.650435 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:20:46.657955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:46.657930 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-f7ad21-74bbc64dff-529c6"] Oct 30 06:20:47.500692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.500603 6171 generic.go:356] "Generic (PLEG): container finished" podID="e8722a33-9b7a-439f-9693-a05cea761700" containerID="8b8ebe65423382c5b9632e4f630db483612b493950dc58ee6b2de752558be3b9" exitCode=0 Oct 30 06:20:47.501193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.500691 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" event={"ID":"e8722a33-9b7a-439f-9693-a05cea761700","Type":"ContainerDied","Data":"8b8ebe65423382c5b9632e4f630db483612b493950dc58ee6b2de752558be3b9"} Oct 30 06:20:47.675967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.675937 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" path="/var/lib/kubelet/pods/3a25279c-bb5e-47fb-9b38-842b88ac29a4/volumes" Oct 30 06:20:47.676554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.676536 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aaf8aff-680a-4c29-a191-854acf6204a0" path="/var/lib/kubelet/pods/8aaf8aff-680a-4c29-a191-854acf6204a0/volumes" Oct 30 06:20:47.677214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.677198 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34" path="/var/lib/kubelet/pods/8fddd0e8-35bb-4b15-a6cf-6f91e5f0bd34/volumes" Oct 30 06:20:47.678161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.678141 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1155157-f3bc-40c7-b0fb-1922161e9196" path="/var/lib/kubelet/pods/b1155157-f3bc-40c7-b0fb-1922161e9196/volumes" Oct 30 06:20:47.678666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:47.678651 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" path="/var/lib/kubelet/pods/d6594fe0-7c2f-4344-9167-0d507b0232ce/volumes" Oct 30 06:20:48.003461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.003435 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:20:48.856602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.856574 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:48.959231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.959192 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\" (UniqueName: \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\") pod \"e8722a33-9b7a-439f-9693-a05cea761700\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " Oct 30 06:20:48.959434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.959314 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\") pod \"e8722a33-9b7a-439f-9693-a05cea761700\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " Oct 30 06:20:48.959595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.959575 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\") pod \"e8722a33-9b7a-439f-9693-a05cea761700\" (UID: \"e8722a33-9b7a-439f-9693-a05cea761700\") " Oct 30 06:20:48.959696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.959663 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca" pod "e8722a33-9b7a-439f-9693-a05cea761700" (UID: "e8722a33-9b7a-439f-9693-a05cea761700"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:48.960216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.960193 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.961645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.961622 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca" pod "e8722a33-9b7a-439f-9693-a05cea761700" (UID: "e8722a33-9b7a-439f-9693-a05cea761700"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:48.961750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:48.961665 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq" pod "e8722a33-9b7a-439f-9693-a05cea761700" (UID: "e8722a33-9b7a-439f-9693-a05cea761700"). InnerVolumeSpecName "kube-api-access-jwhtq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:49.061595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:49.061524 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8722a33-9b7a-439f-9693-a05cea761700-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.061595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:49.061554 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwhtq\" (UniqueName: \"kubernetes.io/projected/e8722a33-9b7a-439f-9693-a05cea761700-kube-api-access-jwhtq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:49.537281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:49.537246 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" Oct 30 06:20:49.537456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:49.537279 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk" event={"ID":"e8722a33-9b7a-439f-9693-a05cea761700","Type":"ContainerDied","Data":"ae07ace37085978790ed9821e07b52d3c6e834cec7215cca0796a8ccfb36c641"} Oct 30 06:20:49.537456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:49.537355 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae07ace37085978790ed9821e07b52d3c6e834cec7215cca0796a8ccfb36c641" Oct 30 06:20:50.247004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.246963 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj"] Oct 30 06:20:50.248122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248097 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8aaf8aff-680a-4c29-a191-854acf6204a0" containerName="payload-pre-processing" Oct 30 06:20:50.248122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248121 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aaf8aff-680a-4c29-a191-854acf6204a0" containerName="payload-pre-processing" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248137 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248142 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7301aaef-11d6-4579-9a2b-a230ea5665f1" containerName="istio-proxy" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248170 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" containerName="payload-processing" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248175 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" containerName="payload-processing" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248187 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e8722a33-9b7a-439f-9693-a05cea761700" containerName="revoke-keys" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248192 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8722a33-9b7a-439f-9693-a05cea761700" containerName="revoke-keys" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248200 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerName="maas-api" Oct 30 06:20:50.248292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248205 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerName="maas-api" Oct 30 06:20:50.248588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248354 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a25279c-bb5e-47fb-9b38-842b88ac29a4" containerName="payload-processing" Oct 30 06:20:50.248588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248365 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8722a33-9b7a-439f-9693-a05cea761700" containerName="revoke-keys" Oct 30 06:20:50.248588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248371 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6594fe0-7c2f-4344-9167-0d507b0232ce" containerName="maas-api" Oct 30 06:20:50.248588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.248387 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aaf8aff-680a-4c29-a191-854acf6204a0" containerName="payload-pre-processing" Oct 30 06:20:50.252456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.252439 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.273223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.273197 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj"] Oct 30 06:20:50.280503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.280460 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:20:50.280627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.280571 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:50.282728 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.282697 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.282847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.282779 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.282847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.282807 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwgv2\" (UniqueName: \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.384037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.384004 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.384037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.384065 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.384258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.384081 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwgv2\" (UniqueName: \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.384829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.384755 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.387401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.387374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.409705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.409679 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwgv2\" (UniqueName: \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\") pod \"maas-api-revoke-sub-4a9918b1-gnddj\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.420284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.420261 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb"] Oct 30 06:20:50.425584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.425563 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.445256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.445230 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb"] Oct 30 06:20:50.486291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.486251 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.486450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.486343 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz4vs\" (UniqueName: \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.486502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.486467 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.567872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.567793 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:50.588324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.588296 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.588450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.588356 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz4vs\" (UniqueName: \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.588450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.588408 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.589271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.589244 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.591203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.591171 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.614554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.614529 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz4vs\" (UniqueName: \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\") pod \"maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.742729 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.742700 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:50.913262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:50.913207 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj"] Oct 30 06:20:50.915625 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:50.915593 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e01b3c7_31bb_429c_a80a_f292b4ce86c2.slice/crio-e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d WatchSource:0}: Error finding container e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d: Status 404 returned error can't find the container with id e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d Oct 30 06:20:51.124009 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:51.123948 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb34485e0_f006_41f9_919c_06010856e90b.slice/crio-261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f WatchSource:0}: Error finding container 261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f: Status 404 returned error can't find the container with id 261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f Oct 30 06:20:51.126087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:51.126052 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb"] Oct 30 06:20:51.569261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:51.569224 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" event={"ID":"7e01b3c7-31bb-429c-a80a-f292b4ce86c2","Type":"ContainerStarted","Data":"0cc02518c8004362e68588a27b589bad5213cd452675711957c34314f48a7962"} Oct 30 06:20:51.569686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:51.569286 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" event={"ID":"7e01b3c7-31bb-429c-a80a-f292b4ce86c2","Type":"ContainerStarted","Data":"e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d"} Oct 30 06:20:51.570812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:51.570783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" event={"ID":"b34485e0-f006-41f9-919c-06010856e90b","Type":"ContainerStarted","Data":"261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f"} Oct 30 06:20:51.614088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:51.614022 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" podStartSLOduration=1.614008111 podStartE2EDuration="1.614008111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:51.612627676 +0000 UTC m=+2910.053004869" watchObservedRunningTime="2026-10-30 06:20:51.614008111 +0000 UTC m=+2910.054385317" Oct 30 06:20:52.581475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:52.581440 6171 generic.go:356] "Generic (PLEG): container finished" podID="7e01b3c7-31bb-429c-a80a-f292b4ce86c2" containerID="0cc02518c8004362e68588a27b589bad5213cd452675711957c34314f48a7962" exitCode=0 Oct 30 06:20:52.581931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:52.581534 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" event={"ID":"7e01b3c7-31bb-429c-a80a-f292b4ce86c2","Type":"ContainerDied","Data":"0cc02518c8004362e68588a27b589bad5213cd452675711957c34314f48a7962"} Oct 30 06:20:52.583256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:52.583232 6171 generic.go:356] "Generic (PLEG): container finished" podID="b34485e0-f006-41f9-919c-06010856e90b" containerID="6e042b71ec72c048d66df38d0c5641ede560c11bede47540a5c0737389c7937a" exitCode=0 Oct 30 06:20:52.583341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:52.583298 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" event={"ID":"b34485e0-f006-41f9-919c-06010856e90b","Type":"ContainerDied","Data":"6e042b71ec72c048d66df38d0c5641ede560c11bede47540a5c0737389c7937a"} Oct 30 06:20:53.923829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.923800 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb"] Oct 30 06:20:53.932256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.932230 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:53.951613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.951584 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:53.964054 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.964015 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb"] Oct 30 06:20:53.995956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.995922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f7nw\" (UniqueName: \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:53.996157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.995982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:53.996157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.996089 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:53.998442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:53.998420 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:54.035849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.035761 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp"] Oct 30 06:20:54.050567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.050536 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83ed428d-7mlcp"] Oct 30 06:20:54.097507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097478 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\") pod \"b34485e0-f006-41f9-919c-06010856e90b\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " Oct 30 06:20:54.097724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097561 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\" (UniqueName: \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\") pod \"b34485e0-f006-41f9-919c-06010856e90b\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " Oct 30 06:20:54.097724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097624 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\") pod \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " Oct 30 06:20:54.097724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097652 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\") pod \"b34485e0-f006-41f9-919c-06010856e90b\" (UID: \"b34485e0-f006-41f9-919c-06010856e90b\") " Oct 30 06:20:54.097910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097841 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\") pod \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " Oct 30 06:20:54.097969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097916 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca" pod "b34485e0-f006-41f9-919c-06010856e90b" (UID: "b34485e0-f006-41f9-919c-06010856e90b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:54.097969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.097933 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\" (UniqueName: \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\") pod \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\" (UID: \"7e01b3c7-31bb-429c-a80a-f292b4ce86c2\") " Oct 30 06:20:54.098525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.098352 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.098525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.098443 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.099114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.098901 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f7nw\" (UniqueName: \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.099114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.099007 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b34485e0-f006-41f9-919c-06010856e90b-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.100651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.100613 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.100830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.100808 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca" pod "7e01b3c7-31bb-429c-a80a-f292b4ce86c2" (UID: "7e01b3c7-31bb-429c-a80a-f292b4ce86c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:54.101170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.101144 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca" pod "7e01b3c7-31bb-429c-a80a-f292b4ce86c2" (UID: "7e01b3c7-31bb-429c-a80a-f292b4ce86c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:54.102302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.102272 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.103060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.103017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2" pod "7e01b3c7-31bb-429c-a80a-f292b4ce86c2" (UID: "7e01b3c7-31bb-429c-a80a-f292b4ce86c2"). InnerVolumeSpecName "kube-api-access-jwgv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:54.103860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.103811 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs" pod "b34485e0-f006-41f9-919c-06010856e90b" (UID: "b34485e0-f006-41f9-919c-06010856e90b"). InnerVolumeSpecName "kube-api-access-sz4vs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:54.105572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.105545 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca" pod "b34485e0-f006-41f9-919c-06010856e90b" (UID: "b34485e0-f006-41f9-919c-06010856e90b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:54.125696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.125672 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f7nw\" (UniqueName: \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.200143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.200075 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sz4vs\" (UniqueName: \"kubernetes.io/projected/b34485e0-f006-41f9-919c-06010856e90b-kube-api-access-sz4vs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.200143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.200105 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.200143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.200115 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b34485e0-f006-41f9-919c-06010856e90b-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.200143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.200123 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.200143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.200131 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwgv2\" (UniqueName: \"kubernetes.io/projected/7e01b3c7-31bb-429c-a80a-f292b4ce86c2-kube-api-access-jwgv2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:54.266165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.266133 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:54.588028 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:54.587994 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ffe99d4_5d63_4a33_962f_6af24f7d2e85.slice/crio-c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22 WatchSource:0}: Error finding container c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22: Status 404 returned error can't find the container with id c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22 Oct 30 06:20:54.591496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.591470 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb"] Oct 30 06:20:54.649299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.649252 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" event={"ID":"8ffe99d4-5d63-4a33-962f-6af24f7d2e85","Type":"ContainerStarted","Data":"c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22"} Oct 30 06:20:54.650898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.650874 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" Oct 30 06:20:54.651076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.650906 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj" event={"ID":"7e01b3c7-31bb-429c-a80a-f292b4ce86c2","Type":"ContainerDied","Data":"e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d"} Oct 30 06:20:54.651076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.650954 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0d4cd52b3f228959bea53d3d88d923aa2c03528bcdbe576a98e1dd17095423d" Oct 30 06:20:54.652516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.652489 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" Oct 30 06:20:54.652638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.652556 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb" event={"ID":"b34485e0-f006-41f9-919c-06010856e90b","Type":"ContainerDied","Data":"261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f"} Oct 30 06:20:54.652638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.652598 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="261cd9663c49bcf08741ed37c36ee5f1aa78c014831990a1e2f339ccf8630e0f" Oct 30 06:20:54.755170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.755135 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:20:54.755354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:54.755335 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" containerID="cri-o://70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00" gracePeriod=30 Oct 30 06:20:55.351862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.351828 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm"] Oct 30 06:20:55.353004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.352981 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b34485e0-f006-41f9-919c-06010856e90b" containerName="revoke-keys" Oct 30 06:20:55.353004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.353001 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b34485e0-f006-41f9-919c-06010856e90b" containerName="revoke-keys" Oct 30 06:20:55.353213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.353094 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7e01b3c7-31bb-429c-a80a-f292b4ce86c2" containerName="revoke-keys" Oct 30 06:20:55.353213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.353102 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e01b3c7-31bb-429c-a80a-f292b4ce86c2" containerName="revoke-keys" Oct 30 06:20:55.353291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.353246 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e01b3c7-31bb-429c-a80a-f292b4ce86c2" containerName="revoke-keys" Oct 30 06:20:55.353291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.353260 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b34485e0-f006-41f9-919c-06010856e90b" containerName="revoke-keys" Oct 30 06:20:55.357190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.357173 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.381213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.381185 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm"] Oct 30 06:20:55.443591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.443550 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.443803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.443615 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.443803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.443765 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njhp4\" (UniqueName: \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.545792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.545753 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.545980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.545812 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.545980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.545853 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njhp4\" (UniqueName: \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.546447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.546425 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.548372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.548355 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.601609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.601575 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njhp4\" (UniqueName: \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.674606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.674578 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:55.679920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.679889 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78a121e4-e83b-42f6-b368-8882fb8f6b21" path="/var/lib/kubelet/pods/78a121e4-e83b-42f6-b368-8882fb8f6b21/volumes" Oct 30 06:20:55.680454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:55.680428 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" event={"ID":"8ffe99d4-5d63-4a33-962f-6af24f7d2e85","Type":"ContainerStarted","Data":"43ab56711b1a77f89b6a433ee96eaaa5716f3adf56d8cc1e42a23b36326a30c2"} Oct 30 06:20:56.029439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.029391 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" podStartSLOduration=3.02937548 podStartE2EDuration="3.02937548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:55.754851572 +0000 UTC m=+2914.195228766" watchObservedRunningTime="2026-10-30 06:20:56.02937548 +0000 UTC m=+2914.469752672" Oct 30 06:20:56.033886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.033858 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 06:20:56.062777 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.062746 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 06:20:56.073331 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:20:56.073301 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197efa0b_0beb_4b7c_867b_01b1b0702d72.slice/crio-aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c WatchSource:0}: Error finding container aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c: Status 404 returned error can't find the container with id aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c Oct 30 06:20:56.074231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.074208 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm"] Oct 30 06:20:56.385121 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:20:56.384992 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:20:56.728889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.728853 6171 generic.go:356] "Generic (PLEG): container finished" podID="8ffe99d4-5d63-4a33-962f-6af24f7d2e85" containerID="43ab56711b1a77f89b6a433ee96eaaa5716f3adf56d8cc1e42a23b36326a30c2" exitCode=0 Oct 30 06:20:56.729281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.729258 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" event={"ID":"8ffe99d4-5d63-4a33-962f-6af24f7d2e85","Type":"ContainerDied","Data":"43ab56711b1a77f89b6a433ee96eaaa5716f3adf56d8cc1e42a23b36326a30c2"} Oct 30 06:20:56.739079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.735510 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" event={"ID":"197efa0b-0beb-4b7c-867b-01b1b0702d72","Type":"ContainerStarted","Data":"d760166655721e0c167673b4b69ba8fcc5deb72e45ec421a2a20fbf4ec8b38e5"} Oct 30 06:20:56.739079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.735900 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" event={"ID":"197efa0b-0beb-4b7c-867b-01b1b0702d72","Type":"ContainerStarted","Data":"aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c"} Oct 30 06:20:56.848376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:56.848301 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" podStartSLOduration=1.848279331 podStartE2EDuration="1.848279331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:56.843667727 +0000 UTC m=+2915.284044916" watchObservedRunningTime="2026-10-30 06:20:56.848279331 +0000 UTC m=+2915.288656526" Oct 30 06:20:57.745776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:57.745743 6171 generic.go:356] "Generic (PLEG): container finished" podID="197efa0b-0beb-4b7c-867b-01b1b0702d72" containerID="d760166655721e0c167673b4b69ba8fcc5deb72e45ec421a2a20fbf4ec8b38e5" exitCode=0 Oct 30 06:20:57.746238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:57.745829 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" event={"ID":"197efa0b-0beb-4b7c-867b-01b1b0702d72","Type":"ContainerDied","Data":"d760166655721e0c167673b4b69ba8fcc5deb72e45ec421a2a20fbf4ec8b38e5"} Oct 30 06:20:57.945247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:57.945219 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:58.037539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.037447 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\") pod \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " Oct 30 06:20:58.037539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.037489 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\") pod \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " Oct 30 06:20:58.037800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.037645 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\" (UniqueName: \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\") pod \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\" (UID: \"8ffe99d4-5d63-4a33-962f-6af24f7d2e85\") " Oct 30 06:20:58.037897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.037873 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca" pod "8ffe99d4-5d63-4a33-962f-6af24f7d2e85" (UID: "8ffe99d4-5d63-4a33-962f-6af24f7d2e85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:58.038394 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.038372 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.040501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.040472 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca" pod "8ffe99d4-5d63-4a33-962f-6af24f7d2e85" (UID: "8ffe99d4-5d63-4a33-962f-6af24f7d2e85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:58.040591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.040519 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw" pod "8ffe99d4-5d63-4a33-962f-6af24f7d2e85" (UID: "8ffe99d4-5d63-4a33-962f-6af24f7d2e85"). InnerVolumeSpecName "kube-api-access-8f7nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:58.140510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.140474 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f7nw\" (UniqueName: \"kubernetes.io/projected/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-kube-api-access-8f7nw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.140510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.140506 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ffe99d4-5d63-4a33-962f-6af24f7d2e85-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:58.756308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.756280 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" Oct 30 06:20:58.756812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.756309 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb" event={"ID":"8ffe99d4-5d63-4a33-962f-6af24f7d2e85","Type":"ContainerDied","Data":"c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22"} Oct 30 06:20:58.756812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.756360 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c308e4e9544819ccf77303debf4e38fac17ec625a38e0b0ce5fd7c1e87a78f22" Oct 30 06:20:58.968212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:58.968181 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:59.053959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.053868 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\") pod \"197efa0b-0beb-4b7c-867b-01b1b0702d72\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " Oct 30 06:20:59.053959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.053913 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\") pod \"197efa0b-0beb-4b7c-867b-01b1b0702d72\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " Oct 30 06:20:59.054231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.054085 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\" (UniqueName: \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\") pod \"197efa0b-0beb-4b7c-867b-01b1b0702d72\" (UID: \"197efa0b-0beb-4b7c-867b-01b1b0702d72\") " Oct 30 06:20:59.054415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.054383 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca" pod "197efa0b-0beb-4b7c-867b-01b1b0702d72" (UID: "197efa0b-0beb-4b7c-867b-01b1b0702d72"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:59.054746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.054721 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.056866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.056840 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca" pod "197efa0b-0beb-4b7c-867b-01b1b0702d72" (UID: "197efa0b-0beb-4b7c-867b-01b1b0702d72"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:59.057073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.056889 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4" pod "197efa0b-0beb-4b7c-867b-01b1b0702d72" (UID: "197efa0b-0beb-4b7c-867b-01b1b0702d72"). InnerVolumeSpecName "kube-api-access-njhp4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:59.156440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.156403 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njhp4\" (UniqueName: \"kubernetes.io/projected/197efa0b-0beb-4b7c-867b-01b1b0702d72-kube-api-access-njhp4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.156440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.156439 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/197efa0b-0beb-4b7c-867b-01b1b0702d72-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:20:59.769911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.769877 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" Oct 30 06:20:59.770361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.769883 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm" event={"ID":"197efa0b-0beb-4b7c-867b-01b1b0702d72","Type":"ContainerDied","Data":"aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c"} Oct 30 06:20:59.770361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:20:59.770144 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa3c470329d1300d061de28e51a6da6052d7e45bf44002b88a3f53992e071c7c" Oct 30 06:21:00.064451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.064416 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:21:00.173981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.173945 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174102 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174227 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174316 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174421 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.174566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174526 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174577 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174603 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174625 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174684 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\") pod \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\" (UID: \"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967\") " Oct 30 06:21:00.174798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.174714 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.175384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.175362 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.176643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.176626 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.176777 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.176312 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.176923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.176465 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:00.177035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.177007 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.179473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.179448 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "kube-api-access-2j5z7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.180560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.180523 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:00.201960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.201909 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.203832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.203790 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy" pod "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" (UID: "a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:00.300307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300233 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300491 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300512 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300527 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300540 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300553 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j5z7\" (UniqueName: \"kubernetes.io/projected/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-kube-api-access-2j5z7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.300644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.300570 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.593012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.592916 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682"] Oct 30 06:21:00.593901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593885 6171 cpu_manager.go:414] "containerMap: removing container" podUID="8ffe99d4-5d63-4a33-962f-6af24f7d2e85" containerName="revoke-keys" Oct 30 06:21:00.593962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593903 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ffe99d4-5d63-4a33-962f-6af24f7d2e85" containerName="revoke-keys" Oct 30 06:21:00.593962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593922 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.593962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593927 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.594104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593975 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.594104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593981 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.594104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593987 6171 cpu_manager.go:414] "containerMap: removing container" podUID="197efa0b-0beb-4b7c-867b-01b1b0702d72" containerName="revoke-keys" Oct 30 06:21:00.594104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.593992 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197efa0b-0beb-4b7c-867b-01b1b0702d72" containerName="revoke-keys" Oct 30 06:21:00.594227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.594141 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.594227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.594150 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="197efa0b-0beb-4b7c-867b-01b1b0702d72" containerName="revoke-keys" Oct 30 06:21:00.594227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.594161 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ffe99d4-5d63-4a33-962f-6af24f7d2e85" containerName="revoke-keys" Oct 30 06:21:00.594227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.594171 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerName="istio-proxy" Oct 30 06:21:00.600236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.600206 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.621148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.621116 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682"] Oct 30 06:21:00.629934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.626835 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:00.629934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.626967 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:21:00.706921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.706862 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.707161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.707072 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwvcv\" (UniqueName: \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.707161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.707140 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.782589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.782552 6171 generic.go:356] "Generic (PLEG): container finished" podID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" containerID="70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00" exitCode=0 Oct 30 06:21:00.783056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.782624 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" Oct 30 06:21:00.783056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.782635 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerDied","Data":"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00"} Oct 30 06:21:00.783056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.782694 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6" event={"ID":"a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967","Type":"ContainerDied","Data":"26b2609c7185ca4ae639f1480a775ddddefbded2db23f918e2e0a36dd22e79a9"} Oct 30 06:21:00.783056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.782717 6171 scope.go:122] "RemoveContainer" containerID="70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00" Oct 30 06:21:00.795953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.795931 6171 scope.go:122] "RemoveContainer" containerID="f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8" Oct 30 06:21:00.808689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.808652 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.808876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.808819 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwvcv\" (UniqueName: \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.809085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.808885 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.810124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.810096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.812400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.812324 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.812652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.812630 6171 scope.go:122] "RemoveContainer" containerID="70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00" Oct 30 06:21:00.813063 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:00.813020 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00\": container with ID starting with 70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00 not found: ID does not exist" containerID="70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00" Oct 30 06:21:00.813171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.813074 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00"} err="failed to get container status \"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00\": rpc error: code = NotFound desc = could not find container \"70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00\": container with ID starting with 70433dee7d0775e51dee0978562b67b0f7c5702f47e57f7b4db6d40125c7cf00 not found: ID does not exist" Oct 30 06:21:00.813171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.813098 6171 scope.go:122] "RemoveContainer" containerID="f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8" Oct 30 06:21:00.813445 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:00.813417 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8\": container with ID starting with f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8 not found: ID does not exist" containerID="f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8" Oct 30 06:21:00.813520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.813453 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8"} err="failed to get container status \"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8\": rpc error: code = NotFound desc = could not find container \"f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8\": container with ID starting with f507f09499f8ed5101ffb5e2a8b07eb9534679b92d3499d4b888a55aa910a3c8 not found: ID does not exist" Oct 30 06:21:00.835719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.835682 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwvcv\" (UniqueName: \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\") pod \"maas-api-revoke-sub-0eb6584f-29682\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:00.845150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.845031 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:21:00.859751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.859714 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-d1bfce-openshift-default-78bfb84cf-qvhb6"] Oct 30 06:21:00.930795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:00.930755 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:01.041093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.034349 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" containerID="cri-o://d47039c82d3ea0c131acfa52505478ad1418906dbd3d1b92c58e37d127594681" gracePeriod=25 Oct 30 06:21:01.063980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.063910 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" containerID="cri-o://c37e289861f68f8052ae47d4c60817980b44d9b6c36d3a6953288d2f6ebd33fd" gracePeriod=25 Oct 30 06:21:01.250008 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:01.249977 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481f57f4_862d_4cd1_85ed_7a78c6994650.slice/crio-01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9 WatchSource:0}: Error finding container 01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9: Status 404 returned error can't find the container with id 01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9 Oct 30 06:21:01.252891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.252849 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682"] Oct 30 06:21:01.292132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.292027 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:21:01.677753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.677671 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967" path="/var/lib/kubelet/pods/a3dbdee0-5c5c-4fc3-a5de-0d0d319f5967/volumes" Oct 30 06:21:01.797924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.797886 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" event={"ID":"481f57f4-862d-4cd1-85ed-7a78c6994650","Type":"ContainerStarted","Data":"c0a6880f9b91b8fe84b0a4883b7b932eceefdb5d0492bdb365c2cdec9cd5dca7"} Oct 30 06:21:01.798408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.797948 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" event={"ID":"481f57f4-862d-4cd1-85ed-7a78c6994650","Type":"ContainerStarted","Data":"01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9"} Oct 30 06:21:01.846979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:01.846916 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" podStartSLOduration=1.8469014719999999 podStartE2EDuration="1.846901472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:01.843023315 +0000 UTC m=+2920.283400509" watchObservedRunningTime="2026-10-30 06:21:01.846901472 +0000 UTC m=+2920.287278662" Oct 30 06:21:02.811940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:02.811902 6171 generic.go:356] "Generic (PLEG): container finished" podID="481f57f4-862d-4cd1-85ed-7a78c6994650" containerID="c0a6880f9b91b8fe84b0a4883b7b932eceefdb5d0492bdb365c2cdec9cd5dca7" exitCode=0 Oct 30 06:21:02.812416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:02.811983 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" event={"ID":"481f57f4-862d-4cd1-85ed-7a78c6994650","Type":"ContainerDied","Data":"c0a6880f9b91b8fe84b0a4883b7b932eceefdb5d0492bdb365c2cdec9cd5dca7"} Oct 30 06:21:03.035736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:03.035695 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp"] Oct 30 06:21:03.041738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:03.041703 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0d15423-7sqkp"] Oct 30 06:21:03.676622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:03.676584 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74d26f15-aa50-418c-95ea-4fa92d9eaf6f" path="/var/lib/kubelet/pods/74d26f15-aa50-418c-95ea-4fa92d9eaf6f/volumes" Oct 30 06:21:04.069075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.069007 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:04.172032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.171998 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:04.172032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.172019 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:04.191146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.191098 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\") pod \"481f57f4-862d-4cd1-85ed-7a78c6994650\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " Oct 30 06:21:04.191369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.191231 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\") pod \"481f57f4-862d-4cd1-85ed-7a78c6994650\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " Oct 30 06:21:04.191536 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.191515 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\" (UniqueName: \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\") pod \"481f57f4-862d-4cd1-85ed-7a78c6994650\" (UID: \"481f57f4-862d-4cd1-85ed-7a78c6994650\") " Oct 30 06:21:04.191633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.191544 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca" pod "481f57f4-862d-4cd1-85ed-7a78c6994650" (UID: "481f57f4-862d-4cd1-85ed-7a78c6994650"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:04.192635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.192607 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:04.194036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.194002 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv" pod "481f57f4-862d-4cd1-85ed-7a78c6994650" (UID: "481f57f4-862d-4cd1-85ed-7a78c6994650"). InnerVolumeSpecName "kube-api-access-gwvcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:04.194510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.194489 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca" pod "481f57f4-862d-4cd1-85ed-7a78c6994650" (UID: "481f57f4-862d-4cd1-85ed-7a78c6994650"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:04.294115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.294001 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/481f57f4-862d-4cd1-85ed-7a78c6994650-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:04.294115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.294035 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwvcv\" (UniqueName: \"kubernetes.io/projected/481f57f4-862d-4cd1-85ed-7a78c6994650-kube-api-access-gwvcv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:04.834905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.834878 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" Oct 30 06:21:04.834905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.834895 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682" event={"ID":"481f57f4-862d-4cd1-85ed-7a78c6994650","Type":"ContainerDied","Data":"01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9"} Oct 30 06:21:04.835246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:04.834948 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01ff9873fa7d37bad7af430f8312f4829c9fec69f692aa012e730d12a42672f9" Oct 30 06:21:05.765708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:05.765666 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:21:05.766363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:05.765887 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" podUID="2944f6bc-3205-419c-962a-0bdf56e52cd1" containerName="payload-processing" containerID="cri-o://ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58" gracePeriod=30 Oct 30 06:21:05.790032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:05.789987 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:21:05.790280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:05.790251 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" podUID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" containerName="payload-pre-processing" containerID="cri-o://b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9" gracePeriod=30 Oct 30 06:21:06.152448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.152419 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:21:06.245080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.239994 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\") pod \"2944f6bc-3205-419c-962a-0bdf56e52cd1\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " Oct 30 06:21:06.245080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.240295 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\") pod \"2944f6bc-3205-419c-962a-0bdf56e52cd1\" (UID: \"2944f6bc-3205-419c-962a-0bdf56e52cd1\") " Oct 30 06:21:06.267591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.255288 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume" pod "2944f6bc-3205-419c-962a-0bdf56e52cd1" (UID: "2944f6bc-3205-419c-962a-0bdf56e52cd1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:06.287069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.275623 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l" pod "2944f6bc-3205-419c-962a-0bdf56e52cd1" (UID: "2944f6bc-3205-419c-962a-0bdf56e52cd1"). InnerVolumeSpecName "kube-api-access-62p9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:06.306700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.306243 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:21:06.350988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.345112 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\") pod \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " Oct 30 06:21:06.350988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.345405 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\" (UniqueName: \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\") pod \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\" (UID: \"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847\") " Oct 30 06:21:06.350988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.346480 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62p9l\" (UniqueName: \"kubernetes.io/projected/2944f6bc-3205-419c-962a-0bdf56e52cd1-kube-api-access-62p9l\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.350988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.346504 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2944f6bc-3205-419c-962a-0bdf56e52cd1-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.362303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.355325 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume" pod "bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" (UID: "bd7e273a-0d9a-4c69-9b6f-e565e8e4f847"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:06.422078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.421766 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6" pod "bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" (UID: "bd7e273a-0d9a-4c69-9b6f-e565e8e4f847"). InnerVolumeSpecName "kube-api-access-rdhr6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:06.453757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.453710 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.453757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.453757 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdhr6\" (UniqueName: \"kubernetes.io/projected/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847-kube-api-access-rdhr6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:06.598518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.598416 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb"] Oct 30 06:21:06.603441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.603410 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-dfc3d2-zjzqb"] Oct 30 06:21:06.647163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.647130 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:21:06.647476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.647449 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" containerID="cri-o://2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428" gracePeriod=30 Oct 30 06:21:06.653677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.653645 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:21:06.654074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:06.654027 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" containerID="cri-o://bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d" gracePeriod=30 Oct 30 06:21:07.039882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.039829 6171 generic.go:356] "Generic (PLEG): container finished" podID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" containerID="b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9" exitCode=0 Oct 30 06:21:07.040473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.039960 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" Oct 30 06:21:07.040473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.039964 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" event={"ID":"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847","Type":"ContainerDied","Data":"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9"} Oct 30 06:21:07.040473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.040023 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2" event={"ID":"bd7e273a-0d9a-4c69-9b6f-e565e8e4f847","Type":"ContainerDied","Data":"9c5238593ef1717cc11225fd1d9c51966ae41d279f198c656b832e193ee23af2"} Oct 30 06:21:07.040473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.040060 6171 scope.go:122] "RemoveContainer" containerID="b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9" Oct 30 06:21:07.047742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.047630 6171 generic.go:356] "Generic (PLEG): container finished" podID="2944f6bc-3205-419c-962a-0bdf56e52cd1" containerID="ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58" exitCode=0 Oct 30 06:21:07.047946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.047775 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" event={"ID":"2944f6bc-3205-419c-962a-0bdf56e52cd1","Type":"ContainerDied","Data":"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58"} Oct 30 06:21:07.047946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.047837 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" event={"ID":"2944f6bc-3205-419c-962a-0bdf56e52cd1","Type":"ContainerDied","Data":"9387a0733a5a449abefc597a33c13daab8c7ef815219d07e5d2f22642a31cbbc"} Oct 30 06:21:07.047946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.047885 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l" Oct 30 06:21:07.060760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.060571 6171 scope.go:122] "RemoveContainer" containerID="b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9" Oct 30 06:21:07.061326 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:07.061095 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9\": container with ID starting with b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9 not found: ID does not exist" containerID="b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9" Oct 30 06:21:07.061326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.061144 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9"} err="failed to get container status \"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9\": rpc error: code = NotFound desc = could not find container \"b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9\": container with ID starting with b6ebf3360ed72c0ef54339f2e9d1f165decfb40db67fd8ae19f8ce6591526df9 not found: ID does not exist" Oct 30 06:21:07.061326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.061169 6171 scope.go:122] "RemoveContainer" containerID="ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58" Oct 30 06:21:07.082238 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.082211 6171 scope.go:122] "RemoveContainer" containerID="ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58" Oct 30 06:21:07.082636 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:07.082607 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58\": container with ID starting with ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58 not found: ID does not exist" containerID="ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58" Oct 30 06:21:07.082711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.082651 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58"} err="failed to get container status \"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58\": rpc error: code = NotFound desc = could not find container \"ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58\": container with ID starting with ea4113a496b654a64835a715a86f480e5e80f654156d1dd7df9912615dd10f58 not found: ID does not exist" Oct 30 06:21:07.120975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.120941 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:21:07.126022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.125988 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-dfc3d2-67fd7bb44c-h6pr2"] Oct 30 06:21:07.162566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.162532 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:21:07.167698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.167663 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-dfc3d2-74d558bbb-d9k2l"] Oct 30 06:21:07.676800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.676760 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2944f6bc-3205-419c-962a-0bdf56e52cd1" path="/var/lib/kubelet/pods/2944f6bc-3205-419c-962a-0bdf56e52cd1/volumes" Oct 30 06:21:07.677395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.677378 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b34485e0-f006-41f9-919c-06010856e90b" path="/var/lib/kubelet/pods/b34485e0-f006-41f9-919c-06010856e90b/volumes" Oct 30 06:21:07.677781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:07.677768 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" path="/var/lib/kubelet/pods/bd7e273a-0d9a-4c69-9b6f-e565e8e4f847/volumes" Oct 30 06:21:11.974836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:11.974806 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:21:11.997908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:11.997880 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:21:12.025461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.025420 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.025692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.025547 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.025692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.025585 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.026833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.026094 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.026833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.026607 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.026833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.026744 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.026936 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.026948 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027004 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027038 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027117 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.027368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027140 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.027782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027540 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.027782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027633 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.030107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.027668 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.028393 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.030227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.028571 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.030227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.028594 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.030227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.028815 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.029346 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "kube-api-access-fk9hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.029617 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.029684 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.029780 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "kube-api-access-ngdq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030278 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030363 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030405 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\") pod \"f52b07bb-95b7-4923-b212-b903da9f71b3\" (UID: \"f52b07bb-95b7-4923-b212-b903da9f71b3\") " Oct 30 06:21:12.030856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030492 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030518 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.030856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030528 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\") pod \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\" (UID: \"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7\") " Oct 30 06:21:12.030856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030709 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.031235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.030992 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert" pod "f52b07bb-95b7-4923-b212-b903da9f71b3" (UID: "f52b07bb-95b7-4923-b212-b903da9f71b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:12.031296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031252 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:12.031341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031323 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:12.031457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031436 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031464 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031479 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f52b07bb-95b7-4923-b212-b903da9f71b3-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031493 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031505 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031519 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fk9hd\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-kube-api-access-fk9hd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031532 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031545 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031557 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031572 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031584 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031597 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdq5\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-kube-api-access-ngdq5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031610 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031623 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.031925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.031635 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f52b07bb-95b7-4923-b212-b903da9f71b3-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.032210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.032189 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:12.032569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.032550 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token" pod "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" (UID: "ec025c0b-1bbe-4792-9b4f-65c55b74d7f7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:12.121635 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.121595 6171 generic.go:356] "Generic (PLEG): container finished" podID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerID="bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d" exitCode=0 Oct 30 06:21:12.121866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.121658 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" Oct 30 06:21:12.121866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.121710 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" event={"ID":"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7","Type":"ContainerDied","Data":"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d"} Oct 30 06:21:12.121866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.121781 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs" event={"ID":"ec025c0b-1bbe-4792-9b4f-65c55b74d7f7","Type":"ContainerDied","Data":"fe7f9ff96e6db24a297370e1803a051a781d66ca4a634a3d97054f6aefa46556"} Oct 30 06:21:12.121866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.121816 6171 scope.go:122] "RemoveContainer" containerID="bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d" Oct 30 06:21:12.124221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.124141 6171 generic.go:356] "Generic (PLEG): container finished" podID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerID="2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428" exitCode=0 Oct 30 06:21:12.124221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.124207 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" Oct 30 06:21:12.124445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.124288 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" event={"ID":"f52b07bb-95b7-4923-b212-b903da9f71b3","Type":"ContainerDied","Data":"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428"} Oct 30 06:21:12.124445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.124334 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2" event={"ID":"f52b07bb-95b7-4923-b212-b903da9f71b3","Type":"ContainerDied","Data":"ed1729b35470b4de18795216a0b9d5d8ccfedc1a1c9ff7d75288969fe8e35c55"} Oct 30 06:21:12.135070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.134013 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.135070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.134068 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.135070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.134084 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:12.139315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.139281 6171 scope.go:122] "RemoveContainer" containerID="bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d" Oct 30 06:21:12.141799 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:12.139666 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d\": container with ID starting with bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d not found: ID does not exist" containerID="bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d" Oct 30 06:21:12.141799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.139697 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d"} err="failed to get container status \"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d\": rpc error: code = NotFound desc = could not find container \"bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d\": container with ID starting with bccefaa46e78b54294338f3579ab8b65b7308025ba7a47e64009d8499c0a595d not found: ID does not exist" Oct 30 06:21:12.141799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.139713 6171 scope.go:122] "RemoveContainer" containerID="2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428" Oct 30 06:21:12.156794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.156765 6171 scope.go:122] "RemoveContainer" containerID="2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428" Oct 30 06:21:12.157191 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:12.157161 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428\": container with ID starting with 2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428 not found: ID does not exist" containerID="2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428" Oct 30 06:21:12.157331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.157203 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428"} err="failed to get container status \"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428\": rpc error: code = NotFound desc = could not find container \"2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428\": container with ID starting with 2639a3c5a304b21d5a89227ab54e472ca74a3ee29eae7ff663b715ee0d4f0428 not found: ID does not exist" Oct 30 06:21:12.219834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.219744 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:21:12.225319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.225279 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-nchbs"] Oct 30 06:21:12.291755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.291709 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:21:12.318926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:12.318881 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-cc33d3-openshift-default-6d67448675-gjjn2"] Oct 30 06:21:13.679131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:13.679076 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" path="/var/lib/kubelet/pods/ec025c0b-1bbe-4792-9b4f-65c55b74d7f7/volumes" Oct 30 06:21:13.679722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:13.679705 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" path="/var/lib/kubelet/pods/f52b07bb-95b7-4923-b212-b903da9f71b3/volumes" Oct 30 06:21:13.960577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:13.960477 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:21:13.960894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:13.960861 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" containerID="cri-o://dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2" gracePeriod=30 Oct 30 06:21:14.273072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:14.271175 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:14.273072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:14.272379 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:19.229475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.229411 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz"] Oct 30 06:21:19.231126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231026 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" Oct 30 06:21:19.231126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231088 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231139 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2944f6bc-3205-419c-962a-0bdf56e52cd1" containerName="payload-processing" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231149 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2944f6bc-3205-419c-962a-0bdf56e52cd1" containerName="payload-processing" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231167 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" containerName="payload-pre-processing" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231175 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" containerName="payload-pre-processing" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231199 6171 cpu_manager.go:414] "containerMap: removing container" podUID="481f57f4-862d-4cd1-85ed-7a78c6994650" containerName="revoke-keys" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231207 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="481f57f4-862d-4cd1-85ed-7a78c6994650" containerName="revoke-keys" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231269 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" Oct 30 06:21:19.231428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231278 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" Oct 30 06:21:19.231922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231559 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="481f57f4-862d-4cd1-85ed-7a78c6994650" containerName="revoke-keys" Oct 30 06:21:19.231922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231589 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f52b07bb-95b7-4923-b212-b903da9f71b3" containerName="istio-proxy" Oct 30 06:21:19.231922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231608 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec025c0b-1bbe-4792-9b4f-65c55b74d7f7" containerName="istio-proxy" Oct 30 06:21:19.231922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231622 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2944f6bc-3205-419c-962a-0bdf56e52cd1" containerName="payload-processing" Oct 30 06:21:19.231922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.231632 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7e273a-0d9a-4c69-9b6f-e565e8e4f847" containerName="payload-pre-processing" Oct 30 06:21:19.237179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.237143 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.252751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.252207 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz"] Oct 30 06:21:19.272473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.272442 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:19.273104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.272890 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:21:19.273559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.273531 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hj9f\" (UniqueName: \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.273699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.273681 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.273890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.273874 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.307034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.306998 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:21:19.335140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.335104 6171 generic.go:356] "Generic (PLEG): container finished" podID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerID="dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2" exitCode=0 Oct 30 06:21:19.335353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.335215 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" Oct 30 06:21:19.335353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.335251 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerDied","Data":"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2"} Oct 30 06:21:19.335353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.335317 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx" event={"ID":"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d","Type":"ContainerDied","Data":"8d6f0e387b0147e51d30efa84e285b1ec938d017742e1fa622752544e3b491c1"} Oct 30 06:21:19.335353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.335347 6171 scope.go:122] "RemoveContainer" containerID="dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2" Oct 30 06:21:19.348127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.348100 6171 scope.go:122] "RemoveContainer" containerID="bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f" Oct 30 06:21:19.372539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.372513 6171 scope.go:122] "RemoveContainer" containerID="dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2" Oct 30 06:21:19.372946 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:19.372923 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2\": container with ID starting with dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2 not found: ID does not exist" containerID="dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2" Oct 30 06:21:19.373030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.372958 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2"} err="failed to get container status \"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2\": rpc error: code = NotFound desc = could not find container \"dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2\": container with ID starting with dea593f611768c8f16f86be063b496f7bfbf0970d9e8681f78d20b07bf90f9f2 not found: ID does not exist" Oct 30 06:21:19.373030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.372979 6171 scope.go:122] "RemoveContainer" containerID="bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f" Oct 30 06:21:19.373417 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:19.373394 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f\": container with ID starting with bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f not found: ID does not exist" containerID="bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f" Oct 30 06:21:19.373482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.373423 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f"} err="failed to get container status \"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f\": rpc error: code = NotFound desc = could not find container \"bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f\": container with ID starting with bc90d9c12a93bd70586fba21e7b5e2f64430ae68a0f2fbf2a1575db7b16e577f not found: ID does not exist" Oct 30 06:21:19.374438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374419 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374506 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374545 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374567 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374694 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374787 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374759 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374868 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374851 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.374952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374897 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:19.374952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.374939 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.375148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375129 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\") pod \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\" (UID: \"fcdd9bf3-ea1c-4b74-a551-b12e83748a7d\") " Oct 30 06:21:19.375293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375271 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:19.375504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375478 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:19.375811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375569 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:19.375944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375817 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:19.376139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.375604 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.376582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.376549 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.377667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.377257 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:19.377667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.377638 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:19.377833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.377540 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.377833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.377690 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:19.377946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.377924 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hj9f\" (UniqueName: \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.378099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378084 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378105 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378118 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378131 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378144 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378157 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378170 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378185 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.378483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378461 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt" pod "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" (UID: "fcdd9bf3-ea1c-4b74-a551-b12e83748a7d"). InnerVolumeSpecName "kube-api-access-92xrt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:19.378902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.378881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.406466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.406434 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hj9f\" (UniqueName: \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\") pod \"maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.479539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.479503 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92xrt\" (UniqueName: \"kubernetes.io/projected/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d-kube-api-access-92xrt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:19.604943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.604867 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:19.705405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.702394 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:21:19.705405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.702447 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-dfc3d2-openshift-default-765b549c9f-b45wx"] Oct 30 06:21:19.923982 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:19.923951 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabf30561_36c9_414b_b9ba_5df0406519cb.slice/crio-b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73 WatchSource:0}: Error finding container b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73: Status 404 returned error can't find the container with id b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73 Oct 30 06:21:19.927514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:19.927489 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz"] Oct 30 06:21:20.346181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:20.346147 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" event={"ID":"abf30561-36c9-414b-b9ba-5df0406519cb","Type":"ContainerStarted","Data":"d3e1a860fc9e8778347f28d81cf8ab202c55dd44075701931664229e553db2b6"} Oct 30 06:21:20.346638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:20.346211 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" event={"ID":"abf30561-36c9-414b-b9ba-5df0406519cb","Type":"ContainerStarted","Data":"b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73"} Oct 30 06:21:20.393596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:20.393542 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" podStartSLOduration=1.393528696 podStartE2EDuration="1.393528696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:20.391930236 +0000 UTC m=+2938.832307429" watchObservedRunningTime="2026-10-30 06:21:20.393528696 +0000 UTC m=+2938.833905910" Oct 30 06:21:21.376216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.376122 6171 generic.go:356] "Generic (PLEG): container finished" podID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerID="d47039c82d3ea0c131acfa52505478ad1418906dbd3d1b92c58e37d127594681" exitCode=0 Oct 30 06:21:21.376904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.376301 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" event={"ID":"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e","Type":"ContainerDied","Data":"d47039c82d3ea0c131acfa52505478ad1418906dbd3d1b92c58e37d127594681"} Oct 30 06:21:21.376904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.376445 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" event={"ID":"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e","Type":"ContainerDied","Data":"ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8"} Oct 30 06:21:21.376904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.376478 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddd577074b0148158267239565036c4d33e6ba8d6ca4c714be9d85c591f937c8" Oct 30 06:21:21.380861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.380827 6171 generic.go:356] "Generic (PLEG): container finished" podID="3517a55f-318e-4335-a017-dd112e066158" containerID="c37e289861f68f8052ae47d4c60817980b44d9b6c36d3a6953288d2f6ebd33fd" exitCode=0 Oct 30 06:21:21.381020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.380898 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" event={"ID":"3517a55f-318e-4335-a017-dd112e066158","Type":"ContainerDied","Data":"c37e289861f68f8052ae47d4c60817980b44d9b6c36d3a6953288d2f6ebd33fd"} Oct 30 06:21:21.381711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.381691 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 06:21:21.383093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.383036 6171 generic.go:356] "Generic (PLEG): container finished" podID="abf30561-36c9-414b-b9ba-5df0406519cb" containerID="d3e1a860fc9e8778347f28d81cf8ab202c55dd44075701931664229e553db2b6" exitCode=0 Oct 30 06:21:21.383210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.383104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" event={"ID":"abf30561-36c9-414b-b9ba-5df0406519cb","Type":"ContainerDied","Data":"d3e1a860fc9e8778347f28d81cf8ab202c55dd44075701931664229e553db2b6"} Oct 30 06:21:21.413392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.413373 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 06:21:21.464582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.464551 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\" (UniqueName: \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\") pod \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " Oct 30 06:21:21.464760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.464602 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\" (UniqueName: \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\") pod \"3517a55f-318e-4335-a017-dd112e066158\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " Oct 30 06:21:21.464760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.464645 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\") pod \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\" (UID: \"a8d300b1-06e7-42f5-afbe-0bdeb7957a9e\") " Oct 30 06:21:21.464903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.464788 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\") pod \"3517a55f-318e-4335-a017-dd112e066158\" (UID: \"3517a55f-318e-4335-a017-dd112e066158\") " Oct 30 06:21:21.465136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.465108 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume" pod "a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" (UID: "a8d300b1-06e7-42f5-afbe-0bdeb7957a9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:21.465389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.465365 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume" pod "3517a55f-318e-4335-a017-dd112e066158" (UID: "3517a55f-318e-4335-a017-dd112e066158"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:21.465861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.465758 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.465861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.465860 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3517a55f-318e-4335-a017-dd112e066158-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.467562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.467539 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh" pod "3517a55f-318e-4335-a017-dd112e066158" (UID: "3517a55f-318e-4335-a017-dd112e066158"). InnerVolumeSpecName "kube-api-access-bw4mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:21.467655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.467575 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z" pod "a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" (UID: "a8d300b1-06e7-42f5-afbe-0bdeb7957a9e"). InnerVolumeSpecName "kube-api-access-mzx6z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:21.567660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.567615 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw4mh\" (UniqueName: \"kubernetes.io/projected/3517a55f-318e-4335-a017-dd112e066158-kube-api-access-bw4mh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.567660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.567644 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzx6z\" (UniqueName: \"kubernetes.io/projected/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e-kube-api-access-mzx6z\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:21.678990 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:21.678918 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" path="/var/lib/kubelet/pods/fcdd9bf3-ea1c-4b74-a551-b12e83748a7d/volumes" Oct 30 06:21:22.023834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.023801 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf"] Oct 30 06:21:22.024648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024631 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.024712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024650 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.024712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024672 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.024712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024678 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.024712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024698 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" Oct 30 06:21:22.024712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024704 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" Oct 30 06:21:22.024874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024737 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" Oct 30 06:21:22.024874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024743 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" Oct 30 06:21:22.024874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024855 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" containerName="payload-processing" Oct 30 06:21:22.024874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024866 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.025002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024875 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdd9bf3-ea1c-4b74-a551-b12e83748a7d" containerName="istio-proxy" Oct 30 06:21:22.025002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.024883 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3517a55f-318e-4335-a017-dd112e066158" containerName="payload-pre-processing" Oct 30 06:21:22.029703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.029683 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.049749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.049725 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf"] Oct 30 06:21:22.074953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.074924 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.075116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.074995 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5dk\" (UniqueName: \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.075237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.075196 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.176896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.176861 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.177038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.176934 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5dk\" (UniqueName: \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.177038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.177009 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.177654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.177618 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.180110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.180085 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.202350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.202328 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5dk\" (UniqueName: \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\") pod \"maas-api-revoke-sub-51398201-cwfkf\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.345750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.345674 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:22.394306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.394276 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" Oct 30 06:21:22.394755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.394299 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv" event={"ID":"3517a55f-318e-4335-a017-dd112e066158","Type":"ContainerDied","Data":"34e1f3444b7f0754d60b765b4c2674aa5dbbca30eccecd271b6d1be15a84803c"} Oct 30 06:21:22.394755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.394371 6171 scope.go:122] "RemoveContainer" containerID="c37e289861f68f8052ae47d4c60817980b44d9b6c36d3a6953288d2f6ebd33fd" Oct 30 06:21:22.394937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.394816 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9" Oct 30 06:21:22.456722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.456117 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 06:21:22.462081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.462035 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-197e9b-6bd8477b49-v6kr9"] Oct 30 06:21:22.515861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.515828 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 06:21:22.524383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.524332 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-197e9b-77c54fd879-kzkzv"] Oct 30 06:21:22.596128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.596074 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:22.666349 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:22.666318 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969fab4d_eea5_4ec9_839c_88a091c2cf2a.slice/crio-6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5 WatchSource:0}: Error finding container 6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5: Status 404 returned error can't find the container with id 6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5 Oct 30 06:21:22.667784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.667755 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf"] Oct 30 06:21:22.684494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.684470 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\" (UniqueName: \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\") pod \"abf30561-36c9-414b-b9ba-5df0406519cb\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " Oct 30 06:21:22.684684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.684630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\") pod \"abf30561-36c9-414b-b9ba-5df0406519cb\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " Oct 30 06:21:22.684684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.684675 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\") pod \"abf30561-36c9-414b-b9ba-5df0406519cb\" (UID: \"abf30561-36c9-414b-b9ba-5df0406519cb\") " Oct 30 06:21:22.685461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.685018 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca" pod "abf30561-36c9-414b-b9ba-5df0406519cb" (UID: "abf30561-36c9-414b-b9ba-5df0406519cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:22.685461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.685403 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:22.687412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.687390 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca" pod "abf30561-36c9-414b-b9ba-5df0406519cb" (UID: "abf30561-36c9-414b-b9ba-5df0406519cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:22.689234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.689215 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f" pod "abf30561-36c9-414b-b9ba-5df0406519cb" (UID: "abf30561-36c9-414b-b9ba-5df0406519cb"). InnerVolumeSpecName "kube-api-access-5hj9f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:22.787411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.787387 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abf30561-36c9-414b-b9ba-5df0406519cb-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:22.787411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:22.787411 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hj9f\" (UniqueName: \"kubernetes.io/projected/abf30561-36c9-414b-b9ba-5df0406519cb-kube-api-access-5hj9f\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:23.406456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.406410 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" event={"ID":"969fab4d-eea5-4ec9-839c-88a091c2cf2a","Type":"ContainerStarted","Data":"6a482c57bf1789c731eb942497c65eb8664149dfc8e03a6db34e831e3890460c"} Oct 30 06:21:23.406932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.406469 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" event={"ID":"969fab4d-eea5-4ec9-839c-88a091c2cf2a","Type":"ContainerStarted","Data":"6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5"} Oct 30 06:21:23.407897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.407875 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" Oct 30 06:21:23.408017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.407912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz" event={"ID":"abf30561-36c9-414b-b9ba-5df0406519cb","Type":"ContainerDied","Data":"b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73"} Oct 30 06:21:23.408017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.407958 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b042af604e0f91472a35c7fcb8254168ba1ff2ee356a082c546d028e998f2d73" Oct 30 06:21:23.459159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.459103 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" podStartSLOduration=2.459086375 podStartE2EDuration="2.459086375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:23.454455545 +0000 UTC m=+2941.894832740" watchObservedRunningTime="2026-10-30 06:21:23.459086375 +0000 UTC m=+2941.899463571" Oct 30 06:21:23.675307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.675248 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3517a55f-318e-4335-a017-dd112e066158" path="/var/lib/kubelet/pods/3517a55f-318e-4335-a017-dd112e066158/volumes" Oct 30 06:21:23.675671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:23.675656 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d300b1-06e7-42f5-afbe-0bdeb7957a9e" path="/var/lib/kubelet/pods/a8d300b1-06e7-42f5-afbe-0bdeb7957a9e/volumes" Oct 30 06:21:24.418537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:24.418502 6171 generic.go:356] "Generic (PLEG): container finished" podID="969fab4d-eea5-4ec9-839c-88a091c2cf2a" containerID="6a482c57bf1789c731eb942497c65eb8664149dfc8e03a6db34e831e3890460c" exitCode=0 Oct 30 06:21:24.418950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:24.418581 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" event={"ID":"969fab4d-eea5-4ec9-839c-88a091c2cf2a","Type":"ContainerDied","Data":"6a482c57bf1789c731eb942497c65eb8664149dfc8e03a6db34e831e3890460c"} Oct 30 06:21:25.600360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.600324 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:25.633857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.633827 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\") pod \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " Oct 30 06:21:25.633996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.633890 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\") pod \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " Oct 30 06:21:25.634100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.634001 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\" (UniqueName: \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\") pod \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\" (UID: \"969fab4d-eea5-4ec9-839c-88a091c2cf2a\") " Oct 30 06:21:25.634211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.634177 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca" pod "969fab4d-eea5-4ec9-839c-88a091c2cf2a" (UID: "969fab4d-eea5-4ec9-839c-88a091c2cf2a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:25.634701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.634664 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:25.636204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.636182 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca" pod "969fab4d-eea5-4ec9-839c-88a091c2cf2a" (UID: "969fab4d-eea5-4ec9-839c-88a091c2cf2a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:25.636298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.636204 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk" pod "969fab4d-eea5-4ec9-839c-88a091c2cf2a" (UID: "969fab4d-eea5-4ec9-839c-88a091c2cf2a"). InnerVolumeSpecName "kube-api-access-dx5dk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:25.736809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.736766 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/969fab4d-eea5-4ec9-839c-88a091c2cf2a-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:25.736809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:25.736798 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5dk\" (UniqueName: \"kubernetes.io/projected/969fab4d-eea5-4ec9-839c-88a091c2cf2a-kube-api-access-dx5dk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:26.037812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:26.037779 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5"] Oct 30 06:21:26.042350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:26.042328 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4187f4a-fsxs5"] Oct 30 06:21:26.383454 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:26.383369 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:21:26.440502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:26.440461 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" event={"ID":"969fab4d-eea5-4ec9-839c-88a091c2cf2a","Type":"ContainerDied","Data":"6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5"} Oct 30 06:21:26.440502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:26.440496 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf" Oct 30 06:21:26.440710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:26.440520 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6ea5c2b61b8646119d3e1ec3aaa696fdaa574a46a45c0297e9070e2d9719acd5" Oct 30 06:21:27.680688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:27.680657 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f59fe6b-a375-46a5-bc76-8185caf669be" path="/var/lib/kubelet/pods/9f59fe6b-a375-46a5-bc76-8185caf669be/volumes" Oct 30 06:21:29.868741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:29.868706 6171 scope.go:122] "RemoveContainer" containerID="d47039c82d3ea0c131acfa52505478ad1418906dbd3d1b92c58e37d127594681" Oct 30 06:21:30.470611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.470573 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2"] Oct 30 06:21:30.472085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472037 6171 cpu_manager.go:414] "containerMap: removing container" podUID="abf30561-36c9-414b-b9ba-5df0406519cb" containerName="revoke-keys" Oct 30 06:21:30.472085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472085 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abf30561-36c9-414b-b9ba-5df0406519cb" containerName="revoke-keys" Oct 30 06:21:30.472254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472107 6171 cpu_manager.go:414] "containerMap: removing container" podUID="969fab4d-eea5-4ec9-839c-88a091c2cf2a" containerName="revoke-keys" Oct 30 06:21:30.472254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472115 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="969fab4d-eea5-4ec9-839c-88a091c2cf2a" containerName="revoke-keys" Oct 30 06:21:30.472351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472330 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="abf30561-36c9-414b-b9ba-5df0406519cb" containerName="revoke-keys" Oct 30 06:21:30.472392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.472356 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="969fab4d-eea5-4ec9-839c-88a091c2cf2a" containerName="revoke-keys" Oct 30 06:21:30.479096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.479077 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.499523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.499495 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2"] Oct 30 06:21:30.505173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.505149 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk"] Oct 30 06:21:30.506619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.505657 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:30.507949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.507923 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:21:30.510417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.510384 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.528308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.528287 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk"] Oct 30 06:21:30.617883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.617854 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.618021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.617886 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqtzw\" (UniqueName: \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.618021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.618004 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.618132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.618071 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.618167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.618151 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.618209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.618195 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdqv\" (UniqueName: \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.719645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719545 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.719645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719576 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqtzw\" (UniqueName: \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.719645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719631 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719677 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719735 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.719794 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdqv\" (UniqueName: \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.720286 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.720390 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.722206 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.722355 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.745190 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdqv\" (UniqueName: \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\") pod \"maas-api-revoke-sub-bc55c08a-ggvwk\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:30.754214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.745229 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqtzw\" (UniqueName: \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\") pod \"maas-api-revoke-sub-45ce7185-dzxd2\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.795993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.795968 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:30.830326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:30.830299 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:31.194682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:31.194643 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2"] Oct 30 06:21:31.195880 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:31.195854 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc673e0a_d7a4_4e70_b918_af7015ca5150.slice/crio-d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278 WatchSource:0}: Error finding container d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278: Status 404 returned error can't find the container with id d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278 Oct 30 06:21:31.228091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:31.228033 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk"] Oct 30 06:21:31.491236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:31.491204 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" event={"ID":"2ea72343-0073-4b22-9931-f3b178d56bac","Type":"ContainerStarted","Data":"7d3d40829f026f9ca2f39eb170e3f95497ae691dac0896db2b0644ed60337c61"} Oct 30 06:21:31.492798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:31.492773 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" event={"ID":"bc673e0a-d7a4-4e70-b918-af7015ca5150","Type":"ContainerStarted","Data":"d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278"} Oct 30 06:21:32.369379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.369342 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv"] Oct 30 06:21:32.374529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.374510 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.391804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.391779 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv"] Oct 30 06:21:32.446219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.446143 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.446375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.446365 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8hqc\" (UniqueName: \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.446509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.446494 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.503849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.503810 6171 generic.go:356] "Generic (PLEG): container finished" podID="bc673e0a-d7a4-4e70-b918-af7015ca5150" containerID="a8a05e85b07e033987958ad5c020d92bc1f3f349d697b106fdb8b77fa8db71b5" exitCode=0 Oct 30 06:21:32.504086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.504036 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" event={"ID":"bc673e0a-d7a4-4e70-b918-af7015ca5150","Type":"ContainerDied","Data":"a8a05e85b07e033987958ad5c020d92bc1f3f349d697b106fdb8b77fa8db71b5"} Oct 30 06:21:32.506251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.506223 6171 generic.go:356] "Generic (PLEG): container finished" podID="2ea72343-0073-4b22-9931-f3b178d56bac" containerID="a33b5cef8b40cb91859147963ce62191d67da9ddff9d08d657929f5daa9efd1d" exitCode=0 Oct 30 06:21:32.506364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.506301 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" event={"ID":"2ea72343-0073-4b22-9931-f3b178d56bac","Type":"ContainerDied","Data":"a33b5cef8b40cb91859147963ce62191d67da9ddff9d08d657929f5daa9efd1d"} Oct 30 06:21:32.548433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.548403 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.548678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.548618 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8hqc\" (UniqueName: \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.548893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.548869 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.549255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.549232 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.551724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.551702 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.573902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.573881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8hqc\" (UniqueName: \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\") pod \"maas-api-revoke-sub-14fa4301-kk5lv\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:32.694192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:32.694163 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:33.043721 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:33.043687 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009a0372_24cb_459e_a1d0_e5ec1f5c1922.slice/crio-33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a WatchSource:0}: Error finding container 33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a: Status 404 returned error can't find the container with id 33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a Oct 30 06:21:33.045420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.045199 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv"] Oct 30 06:21:33.518187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.518150 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" event={"ID":"009a0372-24cb-459e-a1d0-e5ec1f5c1922","Type":"ContainerStarted","Data":"33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a"} Oct 30 06:21:33.772056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.772018 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:33.776171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.776150 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:33.875515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875480 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\" (UniqueName: \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\") pod \"2ea72343-0073-4b22-9931-f3b178d56bac\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " Oct 30 06:21:33.875717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875594 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\") pod \"bc673e0a-d7a4-4e70-b918-af7015ca5150\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " Oct 30 06:21:33.875803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875745 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\") pod \"2ea72343-0073-4b22-9931-f3b178d56bac\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " Oct 30 06:21:33.875867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875813 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\") pod \"bc673e0a-d7a4-4e70-b918-af7015ca5150\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " Oct 30 06:21:33.875936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875887 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\" (UniqueName: \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\") pod \"bc673e0a-d7a4-4e70-b918-af7015ca5150\" (UID: \"bc673e0a-d7a4-4e70-b918-af7015ca5150\") " Oct 30 06:21:33.875997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.875979 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\") pod \"2ea72343-0073-4b22-9931-f3b178d56bac\" (UID: \"2ea72343-0073-4b22-9931-f3b178d56bac\") " Oct 30 06:21:33.876079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.876014 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca" pod "bc673e0a-d7a4-4e70-b918-af7015ca5150" (UID: "bc673e0a-d7a4-4e70-b918-af7015ca5150"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:33.876781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.876753 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.877020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.876991 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca" pod "2ea72343-0073-4b22-9931-f3b178d56bac" (UID: "2ea72343-0073-4b22-9931-f3b178d56bac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:33.879776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.879734 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw" pod "bc673e0a-d7a4-4e70-b918-af7015ca5150" (UID: "bc673e0a-d7a4-4e70-b918-af7015ca5150"). InnerVolumeSpecName "kube-api-access-dqtzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:33.880159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.880134 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca" pod "bc673e0a-d7a4-4e70-b918-af7015ca5150" (UID: "bc673e0a-d7a4-4e70-b918-af7015ca5150"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:33.895300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.895272 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca" pod "2ea72343-0073-4b22-9931-f3b178d56bac" (UID: "2ea72343-0073-4b22-9931-f3b178d56bac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:33.896492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.896464 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv" pod "2ea72343-0073-4b22-9931-f3b178d56bac" (UID: "2ea72343-0073-4b22-9931-f3b178d56bac"). InnerVolumeSpecName "kube-api-access-hmdqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:33.978122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.978098 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqtzw\" (UniqueName: \"kubernetes.io/projected/bc673e0a-d7a4-4e70-b918-af7015ca5150-kube-api-access-dqtzw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.978249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.978122 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.978249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.978139 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdqv\" (UniqueName: \"kubernetes.io/projected/2ea72343-0073-4b22-9931-f3b178d56bac-kube-api-access-hmdqv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.978249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.978153 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ea72343-0073-4b22-9931-f3b178d56bac-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.978249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:33.978162 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bc673e0a-d7a4-4e70-b918-af7015ca5150-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:34.528593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.528564 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" Oct 30 06:21:34.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.528589 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk" event={"ID":"2ea72343-0073-4b22-9931-f3b178d56bac","Type":"ContainerDied","Data":"7d3d40829f026f9ca2f39eb170e3f95497ae691dac0896db2b0644ed60337c61"} Oct 30 06:21:34.529070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.528641 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d3d40829f026f9ca2f39eb170e3f95497ae691dac0896db2b0644ed60337c61" Oct 30 06:21:34.530518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.530488 6171 generic.go:356] "Generic (PLEG): container finished" podID="009a0372-24cb-459e-a1d0-e5ec1f5c1922" containerID="43f43448f56c9f18ba4cbe8cf1ffa97cfc1b27192fd4b50bbeadd5464c578acb" exitCode=0 Oct 30 06:21:34.530667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.530565 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" event={"ID":"009a0372-24cb-459e-a1d0-e5ec1f5c1922","Type":"ContainerDied","Data":"43f43448f56c9f18ba4cbe8cf1ffa97cfc1b27192fd4b50bbeadd5464c578acb"} Oct 30 06:21:34.532189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.532172 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" Oct 30 06:21:34.532189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.532175 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2" event={"ID":"bc673e0a-d7a4-4e70-b918-af7015ca5150","Type":"ContainerDied","Data":"d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278"} Oct 30 06:21:34.532434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:34.532216 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d75350f5b6944756e8916b654b91ab0d9e097c30c8ab1a18a57cbcca3856d278" Oct 30 06:21:35.730875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.730845 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:35.806144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.806024 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\") pod \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " Oct 30 06:21:35.806310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.806246 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\") pod \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " Oct 30 06:21:35.806310 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.806282 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\" (UniqueName: \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\") pod \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\" (UID: \"009a0372-24cb-459e-a1d0-e5ec1f5c1922\") " Oct 30 06:21:35.806511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.806428 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca" pod "009a0372-24cb-459e-a1d0-e5ec1f5c1922" (UID: "009a0372-24cb-459e-a1d0-e5ec1f5c1922"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:35.807073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.807030 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.815469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.815436 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca" pod "009a0372-24cb-459e-a1d0-e5ec1f5c1922" (UID: "009a0372-24cb-459e-a1d0-e5ec1f5c1922"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:35.815600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.815571 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc" pod "009a0372-24cb-459e-a1d0-e5ec1f5c1922" (UID: "009a0372-24cb-459e-a1d0-e5ec1f5c1922"). InnerVolumeSpecName "kube-api-access-v8hqc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:35.908769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.908743 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/009a0372-24cb-459e-a1d0-e5ec1f5c1922-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:35.908769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:35.908769 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8hqc\" (UniqueName: \"kubernetes.io/projected/009a0372-24cb-459e-a1d0-e5ec1f5c1922-kube-api-access-v8hqc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.041981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:36.041954 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8"] Oct 30 06:21:36.047914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:36.047888 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-89c98f1e-rr5k8"] Oct 30 06:21:36.552139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:36.552098 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" event={"ID":"009a0372-24cb-459e-a1d0-e5ec1f5c1922","Type":"ContainerDied","Data":"33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a"} Oct 30 06:21:36.552316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:36.552169 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33edce0d52a1556d87ae311db333fa867690be3bfacf8ec7b0c9adc845b4086a" Oct 30 06:21:36.552316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:36.552112 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv" Oct 30 06:21:37.674338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:37.674302 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca52e85b-1741-4e9d-900a-32c3c5f8c4ed" path="/var/lib/kubelet/pods/ca52e85b-1741-4e9d-900a-32c3c5f8c4ed/volumes" Oct 30 06:21:39.855773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.855738 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh"] Oct 30 06:21:39.856667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856648 6171 cpu_manager.go:414] "containerMap: removing container" podUID="009a0372-24cb-459e-a1d0-e5ec1f5c1922" containerName="revoke-keys" Oct 30 06:21:39.856723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856668 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009a0372-24cb-459e-a1d0-e5ec1f5c1922" containerName="revoke-keys" Oct 30 06:21:39.856723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856684 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2ea72343-0073-4b22-9931-f3b178d56bac" containerName="revoke-keys" Oct 30 06:21:39.856723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856689 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ea72343-0073-4b22-9931-f3b178d56bac" containerName="revoke-keys" Oct 30 06:21:39.856723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856698 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bc673e0a-d7a4-4e70-b918-af7015ca5150" containerName="revoke-keys" Oct 30 06:21:39.856723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856703 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc673e0a-d7a4-4e70-b918-af7015ca5150" containerName="revoke-keys" Oct 30 06:21:39.856888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856822 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="009a0372-24cb-459e-a1d0-e5ec1f5c1922" containerName="revoke-keys" Oct 30 06:21:39.856888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856834 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ea72343-0073-4b22-9931-f3b178d56bac" containerName="revoke-keys" Oct 30 06:21:39.856888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.856841 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc673e0a-d7a4-4e70-b918-af7015ca5150" containerName="revoke-keys" Oct 30 06:21:39.861986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.861965 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:39.881189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.881165 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh"] Oct 30 06:21:39.887601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.887569 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:39.887765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.887659 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:21:39.967588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.967549 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:39.967741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.967673 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:39.967919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:39.967898 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.069263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.069217 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.069446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.069331 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.069446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.069436 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.070178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.070154 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.071987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.071954 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.094906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.094883 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.178387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.178314 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:40.200076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.193482 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:21:40.200076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.193705 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" podUID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" containerName="maas-api" containerID="cri-o://f0ada4bc562a5be6415dbc8f60934f62616617430898740bcbb62d7c5401297b" gracePeriod=30 Oct 30 06:21:40.682848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.682808 6171 generic.go:356] "Generic (PLEG): container finished" podID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" containerID="f0ada4bc562a5be6415dbc8f60934f62616617430898740bcbb62d7c5401297b" exitCode=0 Oct 30 06:21:40.683703 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.683657 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" event={"ID":"da0d3557-e4b6-40b6-bb8c-8f0ab9572196","Type":"ContainerDied","Data":"f0ada4bc562a5be6415dbc8f60934f62616617430898740bcbb62d7c5401297b"} Oct 30 06:21:40.694813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.694780 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh"] Oct 30 06:21:40.821431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.820008 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:21:40.823588 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:40.823561 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c9c110f_bb4a_42e0_ba38_5deb7b5bde5d.slice/crio-db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321 WatchSource:0}: Error finding container db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321: Status 404 returned error can't find the container with id db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321 Oct 30 06:21:40.849343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.849308 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\") pod \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " Oct 30 06:21:40.849480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.849408 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\") pod \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " Oct 30 06:21:40.849711 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.849582 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\" (UniqueName: \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\") pod \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " Oct 30 06:21:40.849804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.849747 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\") pod \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\" (UID: \"da0d3557-e4b6-40b6-bb8c-8f0ab9572196\") " Oct 30 06:21:40.849804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.849781 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca" pod "da0d3557-e4b6-40b6-bb8c-8f0ab9572196" (UID: "da0d3557-e4b6-40b6-bb8c-8f0ab9572196"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:40.850633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.850609 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:40.851901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.851875 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw" pod "da0d3557-e4b6-40b6-bb8c-8f0ab9572196" (UID: "da0d3557-e4b6-40b6-bb8c-8f0ab9572196"). InnerVolumeSpecName "kube-api-access-vxxtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:40.852021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.851918 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls" pod "da0d3557-e4b6-40b6-bb8c-8f0ab9572196" (UID: "da0d3557-e4b6-40b6-bb8c-8f0ab9572196"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:40.852137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.852028 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs" pod "da0d3557-e4b6-40b6-bb8c-8f0ab9572196" (UID: "da0d3557-e4b6-40b6-bb8c-8f0ab9572196"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:40.952457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.952384 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:40.952457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.952414 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxxtw\" (UniqueName: \"kubernetes.io/projected/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-kube-api-access-vxxtw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:40.952457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:40.952423 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da0d3557-e4b6-40b6-bb8c-8f0ab9572196-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:41.445838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.445804 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm"] Oct 30 06:21:41.450419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.450393 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-91f42c-zqklm"] Oct 30 06:21:41.676236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.676203 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197efa0b-0beb-4b7c-867b-01b1b0702d72" path="/var/lib/kubelet/pods/197efa0b-0beb-4b7c-867b-01b1b0702d72/volumes" Oct 30 06:21:41.695904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.695829 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" event={"ID":"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d","Type":"ContainerStarted","Data":"30224cb68a3ff849f38104192d11e67563b0dfc566dc3fc181b9d0ee508c47df"} Oct 30 06:21:41.695904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.695882 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" event={"ID":"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d","Type":"ContainerStarted","Data":"db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321"} Oct 30 06:21:41.697601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.697577 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" Oct 30 06:21:41.697702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.697619 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9" event={"ID":"da0d3557-e4b6-40b6-bb8c-8f0ab9572196","Type":"ContainerDied","Data":"9614e38d3904177a4beb0a21327071f5e6a693008e1209c7775fa50ec03b30fe"} Oct 30 06:21:41.697702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.697680 6171 scope.go:122] "RemoveContainer" containerID="f0ada4bc562a5be6415dbc8f60934f62616617430898740bcbb62d7c5401297b" Oct 30 06:21:41.743080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.743021 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" podStartSLOduration=2.743008234 podStartE2EDuration="2.743008234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:41.74001487 +0000 UTC m=+2960.180392064" watchObservedRunningTime="2026-10-30 06:21:41.743008234 +0000 UTC m=+2960.183385428" Oct 30 06:21:41.787358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.787330 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:21:41.792667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:41.792643 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w5-91f42c-585cd85b46-zj5l9"] Oct 30 06:21:42.709550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:42.709519 6171 generic.go:356] "Generic (PLEG): container finished" podID="2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" containerID="30224cb68a3ff849f38104192d11e67563b0dfc566dc3fc181b9d0ee508c47df" exitCode=0 Oct 30 06:21:42.709959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:42.709611 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" event={"ID":"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d","Type":"ContainerDied","Data":"30224cb68a3ff849f38104192d11e67563b0dfc566dc3fc181b9d0ee508c47df"} Oct 30 06:21:43.719094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:43.718786 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" path="/var/lib/kubelet/pods/da0d3557-e4b6-40b6-bb8c-8f0ab9572196/volumes" Oct 30 06:21:43.952148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:43.952116 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:44.028986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.024709 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\") pod \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " Oct 30 06:21:44.028986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.025111 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\") pod \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " Oct 30 06:21:44.028986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.025302 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\") pod \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\" (UID: \"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d\") " Oct 30 06:21:44.038776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.033124 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb"] Oct 30 06:21:44.038776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.033915 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca" pod "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" (UID: "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:44.039875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.039843 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-477f6b-xzhsb"] Oct 30 06:21:44.043735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.043699 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv" pod "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" (UID: "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d"). InnerVolumeSpecName "kube-api-access-rrdtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:44.045256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.045224 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca" pod "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" (UID: "2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:44.153837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.153806 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.154166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.154145 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.154309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.154286 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrdtv\" (UniqueName: \"kubernetes.io/projected/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d-kube-api-access-rrdtv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.757579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.757554 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" Oct 30 06:21:44.757981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.757553 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh" event={"ID":"2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d","Type":"ContainerDied","Data":"db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321"} Oct 30 06:21:44.757981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:44.757672 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db5a3b1e979948ff847eb3f7b0dbc60a802629c67ccc3307d57de39b3684a321" Oct 30 06:21:45.675884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:45.675844 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ffe99d4-5d63-4a33-962f-6af24f7d2e85" path="/var/lib/kubelet/pods/8ffe99d4-5d63-4a33-962f-6af24f7d2e85/volumes" Oct 30 06:21:48.990192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:48.990142 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:21:48.990905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:48.990876 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" containerID="cri-o://b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2" gracePeriod=30 Oct 30 06:21:49.558952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.558504 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:21:49.707408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.707361 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf"] Oct 30 06:21:49.708316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708295 6171 cpu_manager.go:414] "containerMap: removing container" podUID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" containerName="maas-api" Oct 30 06:21:49.708316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708316 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" containerName="maas-api" Oct 30 06:21:49.708453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708361 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" containerName="revoke-keys" Oct 30 06:21:49.708453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708367 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" containerName="revoke-keys" Oct 30 06:21:49.708543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708507 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0d3557-e4b6-40b6-bb8c-8f0ab9572196" containerName="maas-api" Oct 30 06:21:49.708543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.708521 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" containerName="revoke-keys" Oct 30 06:21:49.713216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.713183 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.730727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.730438 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:49.731157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.731124 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:21:49.735257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.735224 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf"] Oct 30 06:21:49.840250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.840161 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9jwf\" (UniqueName: \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.840250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.840223 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.840469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.840416 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.942186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.942128 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9jwf\" (UniqueName: \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.942410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.942207 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.942410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.942400 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.943485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.943454 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.945637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.945603 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:49.969008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:49.968963 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9jwf\" (UniqueName: \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\") pod \"maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:50.030752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.030709 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:50.253142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.253097 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:21:50.309274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.309239 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:21:50.348797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.348759 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:21:50.367330 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:50.367284 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74b5a263_88ca_46e2_9898_dc12e3139af0.slice/crio-140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4 WatchSource:0}: Error finding container 140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4: Status 404 returned error can't find the container with id 140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4 Oct 30 06:21:50.369622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.369599 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf"] Oct 30 06:21:50.821392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:50.821346 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" event={"ID":"74b5a263-88ca-46e2-9898-dc12e3139af0","Type":"ContainerStarted","Data":"140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4"} Oct 30 06:21:51.837856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:51.837815 6171 generic.go:356] "Generic (PLEG): container finished" podID="74b5a263-88ca-46e2-9898-dc12e3139af0" containerID="dab21ff6aced8f909158366b938ef4d71ea07e6c3c3621da9fb0ed6814222ee8" exitCode=0 Oct 30 06:21:51.838500 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:51.837912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" event={"ID":"74b5a263-88ca-46e2-9898-dc12e3139af0","Type":"ContainerDied","Data":"dab21ff6aced8f909158366b938ef4d71ea07e6c3c3621da9fb0ed6814222ee8"} Oct 30 06:21:53.124454 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.124378 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:21:53.125140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.125105 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:21:53.227097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.227063 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:53.278096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.277622 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 06:21:53.278096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.277823 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" containerID="cri-o://c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4" gracePeriod=30 Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.304438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\" (UniqueName: \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\") pod \"74b5a263-88ca-46e2-9898-dc12e3139af0\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.304763 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\") pod \"74b5a263-88ca-46e2-9898-dc12e3139af0\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.304808 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\") pod \"74b5a263-88ca-46e2-9898-dc12e3139af0\" (UID: \"74b5a263-88ca-46e2-9898-dc12e3139af0\") " Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.305264 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca" pod "74b5a263-88ca-46e2-9898-dc12e3139af0" (UID: "74b5a263-88ca-46e2-9898-dc12e3139af0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.305564 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:53.332100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.314280 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf" pod "74b5a263-88ca-46e2-9898-dc12e3139af0" (UID: "74b5a263-88ca-46e2-9898-dc12e3139af0"). InnerVolumeSpecName "kube-api-access-s9jwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:53.337564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.337536 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca" pod "74b5a263-88ca-46e2-9898-dc12e3139af0" (UID: "74b5a263-88ca-46e2-9898-dc12e3139af0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:53.457116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.452773 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74b5a263-88ca-46e2-9898-dc12e3139af0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:53.457116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.452813 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9jwf\" (UniqueName: \"kubernetes.io/projected/74b5a263-88ca-46e2-9898-dc12e3139af0-kube-api-access-s9jwf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:53.827674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.827642 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc"] Oct 30 06:21:53.828602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.828583 6171 cpu_manager.go:414] "containerMap: removing container" podUID="74b5a263-88ca-46e2-9898-dc12e3139af0" containerName="revoke-keys" Oct 30 06:21:53.828689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.828604 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74b5a263-88ca-46e2-9898-dc12e3139af0" containerName="revoke-keys" Oct 30 06:21:53.828779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.828768 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="74b5a263-88ca-46e2-9898-dc12e3139af0" containerName="revoke-keys" Oct 30 06:21:53.832801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.832777 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.862094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.861711 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc"] Oct 30 06:21:53.862624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.862599 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl68x\" (UniqueName: \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.862746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.862655 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.862815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.862754 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.876087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.874831 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" Oct 30 06:21:53.876087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.874856 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf" event={"ID":"74b5a263-88ca-46e2-9898-dc12e3139af0","Type":"ContainerDied","Data":"140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4"} Oct 30 06:21:53.876087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.874907 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="140d3fe8736940c6579e502549c70f6ea97f1d91abe60c9676a3b41a6025d5e4" Oct 30 06:21:53.964161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.964134 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl68x\" (UniqueName: \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.964323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.964189 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.964323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.964238 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.965021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.964996 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.966760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.966739 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:53.996445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:53.996393 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl68x\" (UniqueName: \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:54.149446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.149415 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:54.298825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.298789 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:21:54.376644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376602 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.376856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376703 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.376856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376752 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.376856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376825 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376895 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.376946 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.377116 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.377156 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.377252 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\") pod \"00ffc362-a4ce-4f99-b0c3-a41834795deb\" (UID: \"00ffc362-a4ce-4f99-b0c3-a41834795deb\") " Oct 30 06:21:54.377810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.377511 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:54.377810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.377764 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.378867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.378833 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.378966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.378871 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.379186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.379160 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.379271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.379193 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.379271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.379209 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.379271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.379223 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/00ffc362-a4ce-4f99-b0c3-a41834795deb-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.379458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.379284 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.380520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.380486 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:54.380637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.380557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:54.380887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.380865 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:54.381402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.381376 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt" pod "00ffc362-a4ce-4f99-b0c3-a41834795deb" (UID: "00ffc362-a4ce-4f99-b0c3-a41834795deb"). InnerVolumeSpecName "kube-api-access-5ljdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:54.462842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.462809 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 06:21:54.473449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.473417 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 06:21:54.481578 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.481482 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ljdt\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-kube-api-access-5ljdt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.481779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.481757 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.481874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.481785 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.481874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.481800 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/00ffc362-a4ce-4f99-b0c3-a41834795deb-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.481874 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.481819 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/00ffc362-a4ce-4f99-b0c3-a41834795deb-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:54.487853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.487804 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 06:21:54.518225 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:21:54.518191 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod822d3556_d424_438e_a4c8_54282e213241.slice/crio-447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e WatchSource:0}: Error finding container 447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e: Status 404 returned error can't find the container with id 447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e Oct 30 06:21:54.519841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.519717 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc"] Oct 30 06:21:54.885627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.885590 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" event={"ID":"822d3556-d424-438e-a4c8-54282e213241","Type":"ContainerStarted","Data":"738908b890c0d0e26647a4908867053bf8ed3075eec8dd9170f9152d0b7dbe67"} Oct 30 06:21:54.885758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.885649 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" event={"ID":"822d3556-d424-438e-a4c8-54282e213241","Type":"ContainerStarted","Data":"447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e"} Oct 30 06:21:54.887090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.887063 6171 generic.go:356] "Generic (PLEG): container finished" podID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerID="b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2" exitCode=0 Oct 30 06:21:54.887210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.887099 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" Oct 30 06:21:54.887210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.887135 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" event={"ID":"00ffc362-a4ce-4f99-b0c3-a41834795deb","Type":"ContainerDied","Data":"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2"} Oct 30 06:21:54.887210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.887176 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq" event={"ID":"00ffc362-a4ce-4f99-b0c3-a41834795deb","Type":"ContainerDied","Data":"b307b2b84a5adb4248312debdcdad3174465bffc7aedce7f9acd69598f8f1cc5"} Oct 30 06:21:54.887210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.887197 6171 scope.go:122] "RemoveContainer" containerID="b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2" Oct 30 06:21:54.900428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.900413 6171 scope.go:122] "RemoveContainer" containerID="b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2" Oct 30 06:21:54.900687 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:54.900668 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2\": container with ID starting with b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2 not found: ID does not exist" containerID="b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2" Oct 30 06:21:54.900760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.900698 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2"} err="failed to get container status \"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2\": rpc error: code = NotFound desc = could not find container \"b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2\": container with ID starting with b5523b613d5f9233d367060ab8a79605f3185495a203e28086cef691f5276ce2 not found: ID does not exist" Oct 30 06:21:54.953701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:54.953657 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" podStartSLOduration=1.953645013 podStartE2EDuration="1.953645013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:54.950974993 +0000 UTC m=+2973.391352202" watchObservedRunningTime="2026-10-30 06:21:54.953645013 +0000 UTC m=+2973.394022236" Oct 30 06:21:55.010013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.009983 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:21:55.015286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.015260 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-91f42c-openshift-default-b975d7965-htglq"] Oct 30 06:21:55.309769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.309736 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" containerID="cri-o://fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246" gracePeriod=25 Oct 30 06:21:55.349123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.349089 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" containerID="cri-o://c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581" gracePeriod=25 Oct 30 06:21:55.675173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.675102 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" path="/var/lib/kubelet/pods/00ffc362-a4ce-4f99-b0c3-a41834795deb/volumes" Oct 30 06:21:55.898736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.898698 6171 generic.go:356] "Generic (PLEG): container finished" podID="822d3556-d424-438e-a4c8-54282e213241" containerID="738908b890c0d0e26647a4908867053bf8ed3075eec8dd9170f9152d0b7dbe67" exitCode=0 Oct 30 06:21:55.898914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:55.898784 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" event={"ID":"822d3556-d424-438e-a4c8-54282e213241","Type":"ContainerDied","Data":"738908b890c0d0e26647a4908867053bf8ed3075eec8dd9170f9152d0b7dbe67"} Oct 30 06:21:56.381703 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:56.381668 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:21:57.080169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.080143 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:57.121377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.121342 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\") pod \"822d3556-d424-438e-a4c8-54282e213241\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " Oct 30 06:21:57.121740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.121714 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\") pod \"822d3556-d424-438e-a4c8-54282e213241\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " Oct 30 06:21:57.121740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.121726 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca" pod "822d3556-d424-438e-a4c8-54282e213241" (UID: "822d3556-d424-438e-a4c8-54282e213241"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:57.121895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.121780 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\" (UniqueName: \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\") pod \"822d3556-d424-438e-a4c8-54282e213241\" (UID: \"822d3556-d424-438e-a4c8-54282e213241\") " Oct 30 06:21:57.122465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.122380 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/822d3556-d424-438e-a4c8-54282e213241-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.124423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.124142 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca" pod "822d3556-d424-438e-a4c8-54282e213241" (UID: "822d3556-d424-438e-a4c8-54282e213241"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:57.124423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.124852 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x" pod "822d3556-d424-438e-a4c8-54282e213241" (UID: "822d3556-d424-438e-a4c8-54282e213241"). InnerVolumeSpecName "kube-api-access-xl68x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:57.223774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.223749 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/822d3556-d424-438e-a4c8-54282e213241-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.223909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.223775 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl68x\" (UniqueName: \"kubernetes.io/projected/822d3556-d424-438e-a4c8-54282e213241-kube-api-access-xl68x\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:57.920544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.920510 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" Oct 30 06:21:57.920544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.920539 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc" event={"ID":"822d3556-d424-438e-a4c8-54282e213241","Type":"ContainerDied","Data":"447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e"} Oct 30 06:21:57.921012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:57.920589 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="447df9e9420978679cc765876576dad99cf82c74371846f631ddfbf1c166ec1e" Oct 30 06:21:58.490272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.490244 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 06:21:58.548796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.548740 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.548796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.548774 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.548981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.548825 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.548981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.548859 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.548981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.548978 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.549193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549019 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:58.549193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549085 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.549193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549129 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:58.549193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549162 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.549425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549255 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.549425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549273 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:58.549425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549326 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\") pod \"961d6e36-ecec-4957-abdc-76ea72000259\" (UID: \"961d6e36-ecec-4957-abdc-76ea72000259\") " Oct 30 06:21:58.549425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549378 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:58.549807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549521 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:58.549903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549885 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.549979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549907 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961d6e36-ecec-4957-abdc-76ea72000259-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.549979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549916 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.549979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549926 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.549979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.549934 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.551401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.551377 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:58.551497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.551477 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "kube-api-access-hr5wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:58.551802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.551770 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:58.551900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.551832 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy" pod "961d6e36-ecec-4957-abdc-76ea72000259" (UID: "961d6e36-ecec-4957-abdc-76ea72000259"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:58.651781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.651753 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961d6e36-ecec-4957-abdc-76ea72000259-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.651949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.651781 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961d6e36-ecec-4957-abdc-76ea72000259-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.651949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.651829 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hr5wh\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-kube-api-access-hr5wh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.651949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.651839 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961d6e36-ecec-4957-abdc-76ea72000259-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:21:58.932036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.931946 6171 generic.go:356] "Generic (PLEG): container finished" podID="961d6e36-ecec-4957-abdc-76ea72000259" containerID="c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4" exitCode=0 Oct 30 06:21:58.932036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.931993 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" Oct 30 06:21:58.932036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.932026 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerDied","Data":"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4"} Oct 30 06:21:58.932683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.932092 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l" event={"ID":"961d6e36-ecec-4957-abdc-76ea72000259","Type":"ContainerDied","Data":"655f505cad4d9176d16ca9357ee52bd47763f8f737557042cc7d697af95c4308"} Oct 30 06:21:58.932683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.932114 6171 scope.go:122] "RemoveContainer" containerID="c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4" Oct 30 06:21:58.944507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.944491 6171 scope.go:122] "RemoveContainer" containerID="185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008" Oct 30 06:21:58.954955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.954936 6171 scope.go:122] "RemoveContainer" containerID="c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4" Oct 30 06:21:58.955233 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:58.955209 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4\": container with ID starting with c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4 not found: ID does not exist" containerID="c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4" Oct 30 06:21:58.955329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.955237 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4"} err="failed to get container status \"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4\": rpc error: code = NotFound desc = could not find container \"c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4\": container with ID starting with c4182c7de911cb9d2044bf39b2d4db06b56a42e42491710e372cbab02f46e2c4 not found: ID does not exist" Oct 30 06:21:58.955329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.955256 6171 scope.go:122] "RemoveContainer" containerID="185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008" Oct 30 06:21:58.955587 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:21:58.955561 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008\": container with ID starting with 185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008 not found: ID does not exist" containerID="185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008" Oct 30 06:21:58.955655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.955598 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008"} err="failed to get container status \"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008\": rpc error: code = NotFound desc = could not find container \"185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008\": container with ID starting with 185a9a1bbc3fba4af4c4d66595ed2f811cb533ee9a66f1188125743ffb597008 not found: ID does not exist" Oct 30 06:21:58.970146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.970123 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn_9d014002-9bed-4379-aea2-89b267686ae5/payload-processing/0.log" Oct 30 06:21:58.992660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:58.992641 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 06:21:59.006994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:59.006962 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-477f6b-openshift-default-7854c78755-g864l"] Oct 30 06:21:59.162175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:59.162155 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:21:59.353404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:59.353384 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:21:59.586904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:59.586867 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:59.674642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:21:59.674572 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961d6e36-ecec-4957-abdc-76ea72000259" path="/var/lib/kubelet/pods/961d6e36-ecec-4957-abdc-76ea72000259/volumes" Oct 30 06:22:00.156265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.156231 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 06:22:00.206804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.206773 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 06:22:00.298679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.298641 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs"] Oct 30 06:22:00.299922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.299896 6171 cpu_manager.go:414] "containerMap: removing container" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.300141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300123 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.300280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300255 6171 cpu_manager.go:414] "containerMap: removing container" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.300280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300270 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.300512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300350 6171 cpu_manager.go:414] "containerMap: removing container" podUID="822d3556-d424-438e-a4c8-54282e213241" containerName="revoke-keys" Oct 30 06:22:00.300512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300359 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="822d3556-d424-438e-a4c8-54282e213241" containerName="revoke-keys" Oct 30 06:22:00.300512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300399 6171 cpu_manager.go:414] "containerMap: removing container" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" Oct 30 06:22:00.300512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300408 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" Oct 30 06:22:00.300774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300618 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="822d3556-d424-438e-a4c8-54282e213241" containerName="revoke-keys" Oct 30 06:22:00.300774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300637 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ffc362-a4ce-4f99-b0c3-a41834795deb" containerName="istio-proxy" Oct 30 06:22:00.300774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300651 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.300774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.300662 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="961d6e36-ecec-4957-abdc-76ea72000259" containerName="istio-proxy" Oct 30 06:22:00.316301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.316267 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.328532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.328139 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs"] Oct 30 06:22:00.334923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.334243 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:00.334923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.334342 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:22:00.382155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.382115 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.382518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.382491 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.382871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.382848 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4xjf\" (UniqueName: \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.484713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.484671 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.484896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.484870 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4xjf\" (UniqueName: \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.484970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.484954 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.485661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.485583 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.487316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.487295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.510792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.510762 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4xjf\" (UniqueName: \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\") pod \"maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:00.599452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.599406 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:00.639375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:00.639346 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:01.003572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.003109 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs"] Oct 30 06:22:01.431124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.431007 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:22:01.431551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.431273 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" podUID="e4a02500-dd1d-49cd-a510-6b93254bafac" containerName="payload-pre-processing" containerID="cri-o://6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7" gracePeriod=30 Oct 30 06:22:01.745828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.745798 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:22:01.867117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.832616 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\" (UniqueName: \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\") pod \"e4a02500-dd1d-49cd-a510-6b93254bafac\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " Oct 30 06:22:01.867117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.833071 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\") pod \"e4a02500-dd1d-49cd-a510-6b93254bafac\" (UID: \"e4a02500-dd1d-49cd-a510-6b93254bafac\") " Oct 30 06:22:01.867117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.855754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume" pod "e4a02500-dd1d-49cd-a510-6b93254bafac" (UID: "e4a02500-dd1d-49cd-a510-6b93254bafac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:01.867117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.862323 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p" pod "e4a02500-dd1d-49cd-a510-6b93254bafac" (UID: "e4a02500-dd1d-49cd-a510-6b93254bafac"). InnerVolumeSpecName "kube-api-access-7sm8p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:01.956483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.936215 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4a02500-dd1d-49cd-a510-6b93254bafac-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:01.956483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:01.936257 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sm8p\" (UniqueName: \"kubernetes.io/projected/e4a02500-dd1d-49cd-a510-6b93254bafac-kube-api-access-7sm8p\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.003795 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" event={"ID":"d2de5d50-1858-4ed6-905b-c563cd576a57","Type":"ContainerStarted","Data":"7dbb380db7e2e9ebce1abf2685a3ba3630ffaf4a260cd714d5ccf8b5f5156a9e"} Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.003873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" event={"ID":"d2de5d50-1858-4ed6-905b-c563cd576a57","Type":"ContainerStarted","Data":"1da2d4b583d758404ea30dca5ff89fef1b33f0d717963644e54219c91a4f59e5"} Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.007240 6171 generic.go:356] "Generic (PLEG): container finished" podID="e4a02500-dd1d-49cd-a510-6b93254bafac" containerID="6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7" exitCode=0 Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.007436 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" event={"ID":"e4a02500-dd1d-49cd-a510-6b93254bafac","Type":"ContainerDied","Data":"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7"} Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.007479 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" event={"ID":"e4a02500-dd1d-49cd-a510-6b93254bafac","Type":"ContainerDied","Data":"2dc3781605ff0a546c1aa86665d4cd27e4d1a1cf75594ac59b79fd7c92f69982"} Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.007502 6171 scope.go:122] "RemoveContainer" containerID="6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7" Oct 30 06:22:02.013592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.007632 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr" Oct 30 06:22:02.035586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.035464 6171 scope.go:122] "RemoveContainer" containerID="6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7" Oct 30 06:22:02.036140 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:02.036032 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7\": container with ID starting with 6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7 not found: ID does not exist" containerID="6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7" Oct 30 06:22:02.036140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.036088 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7"} err="failed to get container status \"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7\": rpc error: code = NotFound desc = could not find container \"6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7\": container with ID starting with 6c1fbbf27fcfa874ec1b07e4f177e10b0f0760321b2064673858d618fc933cb7 not found: ID does not exist" Oct 30 06:22:02.057653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.057598 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" podStartSLOduration=2.057583588 podStartE2EDuration="2.057583588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:02.052659258 +0000 UTC m=+2980.493036451" watchObservedRunningTime="2026-10-30 06:22:02.057583588 +0000 UTC m=+2980.497960792" Oct 30 06:22:02.097909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.097873 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:22:02.103458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.103427 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5deb36-5544b6dfdc-zdzzr"] Oct 30 06:22:02.494823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.494774 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz"] Oct 30 06:22:02.507489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:02.507453 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5deb36-4qhbz"] Oct 30 06:22:03.018013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:03.017976 6171 generic.go:356] "Generic (PLEG): container finished" podID="d2de5d50-1858-4ed6-905b-c563cd576a57" containerID="7dbb380db7e2e9ebce1abf2685a3ba3630ffaf4a260cd714d5ccf8b5f5156a9e" exitCode=0 Oct 30 06:22:03.018232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:03.018080 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" event={"ID":"d2de5d50-1858-4ed6-905b-c563cd576a57","Type":"ContainerDied","Data":"7dbb380db7e2e9ebce1abf2685a3ba3630ffaf4a260cd714d5ccf8b5f5156a9e"} Oct 30 06:22:03.674460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:03.674430 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abf30561-36c9-414b-b9ba-5df0406519cb" path="/var/lib/kubelet/pods/abf30561-36c9-414b-b9ba-5df0406519cb/volumes" Oct 30 06:22:03.674969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:03.674953 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4a02500-dd1d-49cd-a510-6b93254bafac" path="/var/lib/kubelet/pods/e4a02500-dd1d-49cd-a510-6b93254bafac/volumes" Oct 30 06:22:04.204748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.204718 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:04.241239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.241209 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 06:22:04.274483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.274450 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\" (UniqueName: \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\") pod \"d2de5d50-1858-4ed6-905b-c563cd576a57\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " Oct 30 06:22:04.274654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.274535 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\") pod \"d2de5d50-1858-4ed6-905b-c563cd576a57\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " Oct 30 06:22:04.274654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.274598 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\") pod \"d2de5d50-1858-4ed6-905b-c563cd576a57\" (UID: \"d2de5d50-1858-4ed6-905b-c563cd576a57\") " Oct 30 06:22:04.274977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.274949 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca" pod "d2de5d50-1858-4ed6-905b-c563cd576a57" (UID: "d2de5d50-1858-4ed6-905b-c563cd576a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:04.275371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.275346 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:04.276864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.276832 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf" pod "d2de5d50-1858-4ed6-905b-c563cd576a57" (UID: "d2de5d50-1858-4ed6-905b-c563cd576a57"). InnerVolumeSpecName "kube-api-access-b4xjf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:04.277002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.276922 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca" pod "d2de5d50-1858-4ed6-905b-c563cd576a57" (UID: "d2de5d50-1858-4ed6-905b-c563cd576a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:04.377683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.377647 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2de5d50-1858-4ed6-905b-c563cd576a57-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:04.377683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.377684 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4xjf\" (UniqueName: \"kubernetes.io/projected/d2de5d50-1858-4ed6-905b-c563cd576a57-kube-api-access-b4xjf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:04.601712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:04.601621 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" containerID="cri-o://9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b" gracePeriod=45 Oct 30 06:22:05.019158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.019127 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:22:05.041543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.041499 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" event={"ID":"d2de5d50-1858-4ed6-905b-c563cd576a57","Type":"ContainerDied","Data":"1da2d4b583d758404ea30dca5ff89fef1b33f0d717963644e54219c91a4f59e5"} Oct 30 06:22:05.044113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.041535 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs" Oct 30 06:22:05.044113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044036 6171 generic.go:356] "Generic (PLEG): container finished" podID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerID="9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b" exitCode=0 Oct 30 06:22:05.044349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044196 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" Oct 30 06:22:05.044508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044474 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1da2d4b583d758404ea30dca5ff89fef1b33f0d717963644e54219c91a4f59e5" Oct 30 06:22:05.044508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044516 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" event={"ID":"c0ab1062-bade-46bb-906a-6e92a09d12f1","Type":"ContainerDied","Data":"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b"} Oct 30 06:22:05.044753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044541 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp" event={"ID":"c0ab1062-bade-46bb-906a-6e92a09d12f1","Type":"ContainerDied","Data":"38b594f2ecd7fbc80c71a2e206a6409206bc41cb34b0a0b7cf76451ef5a5b218"} Oct 30 06:22:05.044753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.044564 6171 scope.go:122] "RemoveContainer" containerID="9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b" Oct 30 06:22:05.061221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.061202 6171 scope.go:122] "RemoveContainer" containerID="9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b" Oct 30 06:22:05.061516 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:05.061494 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b\": container with ID starting with 9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b not found: ID does not exist" containerID="9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b" Oct 30 06:22:05.061586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.061525 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b"} err="failed to get container status \"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b\": rpc error: code = NotFound desc = could not find container \"9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b\": container with ID starting with 9b86a7c4e9dcdfc09130f54a0dd5cecd5ec5225a530e7caf1bb6fb023f7b2f8b not found: ID does not exist" Oct 30 06:22:05.139030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.138995 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139111 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139172 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\" (UniqueName: \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139263 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139296 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139518 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139382 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.139607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139570 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.139607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139595 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\") pod \"c0ab1062-bade-46bb-906a-6e92a09d12f1\" (UID: \"c0ab1062-bade-46bb-906a-6e92a09d12f1\") " Oct 30 06:22:05.139763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.139603 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.140332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.140307 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.140332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.140334 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.140546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.140355 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.141910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.141873 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "kube-api-access-zsv4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:05.142212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.142189 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.142212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.142198 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs" pod "c0ab1062-bade-46bb-906a-6e92a09d12f1" (UID: "c0ab1062-bade-46bb-906a-6e92a09d12f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:05.157239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.157206 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" containerID="cri-o://0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c" gracePeriod=25 Oct 30 06:22:05.177253 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.177214 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:05.207421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.207391 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" containerID="cri-o://be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4" gracePeriod=25 Oct 30 06:22:05.242015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.241984 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c0ab1062-bade-46bb-906a-6e92a09d12f1-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.242015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.242012 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c0ab1062-bade-46bb-906a-6e92a09d12f1-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.242150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.242027 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsv4g\" (UniqueName: \"kubernetes.io/projected/c0ab1062-bade-46bb-906a-6e92a09d12f1-kube-api-access-zsv4g\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.402175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.402145 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:22:05.407228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.407202 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-25d881/auth-test-model-25d881-kserve-77cb5b78fc-5dxmp"] Oct 30 06:22:05.674665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.674581 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" path="/var/lib/kubelet/pods/c0ab1062-bade-46bb-906a-6e92a09d12f1/volumes" Oct 30 06:22:05.832316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.832282 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:22:05.832559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:05.832532 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" podUID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" containerName="maas-api" containerID="cri-o://e87dd7fdc8950867d75bc75c0234805aa29957cd0cb54d6d5a9c331fdb1fde96" gracePeriod=30 Oct 30 06:22:06.061087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.058751 6171 generic.go:356] "Generic (PLEG): container finished" podID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" containerID="e87dd7fdc8950867d75bc75c0234805aa29957cd0cb54d6d5a9c331fdb1fde96" exitCode=0 Oct 30 06:22:06.061087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.058829 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" event={"ID":"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6","Type":"ContainerDied","Data":"e87dd7fdc8950867d75bc75c0234805aa29957cd0cb54d6d5a9c331fdb1fde96"} Oct 30 06:22:06.141724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.141694 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:22:06.166479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.166454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\") pod \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " Oct 30 06:22:06.166611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.166534 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\") pod \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " Oct 30 06:22:06.166611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.166594 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\") pod \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " Oct 30 06:22:06.166769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.166756 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\" (UniqueName: \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\") pod \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\" (UID: \"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6\") " Oct 30 06:22:06.166831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.166786 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca" pod "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" (UID: "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:06.167371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.167275 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:06.168797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.168774 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls" pod "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" (UID: "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:06.168877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.168838 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j" pod "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" (UID: "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6"). InnerVolumeSpecName "kube-api-access-9sd9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:06.168877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.168839 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs" pod "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" (UID: "d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:06.268449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.268418 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sd9j\" (UniqueName: \"kubernetes.io/projected/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-kube-api-access-9sd9j\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:06.268449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.268447 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:06.268449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.268455 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:06.615965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.615887 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh"] Oct 30 06:22:06.623574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:06.623551 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-24db22-kcnfh"] Oct 30 06:22:07.078446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.078413 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" Oct 30 06:22:07.078970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.078444 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb" event={"ID":"d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6","Type":"ContainerDied","Data":"3aadbd20ef9744f9f88127e1074c8a93851b921ca40eb91ebfef4ce4330b3983"} Oct 30 06:22:07.078970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.078512 6171 scope.go:122] "RemoveContainer" containerID="e87dd7fdc8950867d75bc75c0234805aa29957cd0cb54d6d5a9c331fdb1fde96" Oct 30 06:22:07.139701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.139665 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:22:07.144771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.144749 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w1-24db22-5d8467f488-tjbdb"] Oct 30 06:22:07.676721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.676685 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d" path="/var/lib/kubelet/pods/2c9c110f-bb4a-42e0-ba38-5deb7b5bde5d/volumes" Oct 30 06:22:07.677346 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:07.677330 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" path="/var/lib/kubelet/pods/d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6/volumes" Oct 30 06:22:09.241532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.241474 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" podUID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" containerName="payload-processing" containerID="cri-o://248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa" gracePeriod=25 Oct 30 06:22:09.490746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.490650 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" containerID="cri-o://63f583613e79a264dd2b9e48001076a99f1f78ff87bf4ad259279725ba17f197" gracePeriod=45 Oct 30 06:22:09.501760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.501563 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" containerID="cri-o://6561fb6a0bf2e14303dd7191d446e5e5417a8b4e33fae51cb599e6b8fc6280dc" gracePeriod=45 Oct 30 06:22:09.511734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.510938 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" containerID="cri-o://9bf916eb5df7e7548fa92a20ccd90bbbcaa8a1f495d14fb509758a72f77fd277" gracePeriod=45 Oct 30 06:22:09.587438 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.587409 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:09.615119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.609291 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" probeResult="failure" output="Get \"https://10.133.0.86:8000/ready\": dial tcp 10.133.0.86:8000: connect: connection refused" Oct 30 06:22:09.778995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.778899 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:22:09.779583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.779511 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" containerID="cri-o://9bd63cfed53ee35b54a37c48bbbc2bd9e5ce5363b699b1839aceec09aaacc010" gracePeriod=30 Oct 30 06:22:09.799298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.799267 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:22:09.800484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.799501 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" containerID="cri-o://8fa59c1d018244ae21cc2e4bcc6a804f8effd34700ae432b3f2ca965b808e37d" gracePeriod=30 Oct 30 06:22:09.812994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.812703 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:22:09.812994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.812959 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" containerID="cri-o://d8bc8392161b62ae3f512013c0100fd97b527c42c7c4df16d4098c26381d8392" gracePeriod=30 Oct 30 06:22:09.822315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:09.822204 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 06:22:10.024271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.012974 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\") pod \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " Oct 30 06:22:10.024271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.013374 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\" (UniqueName: \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\") pod \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\" (UID: \"00c7e4b6-b102-4e58-9ab8-04df57b6b1df\") " Oct 30 06:22:10.024271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.023294 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume" pod "00c7e4b6-b102-4e58-9ab8-04df57b6b1df" (UID: "00c7e4b6-b102-4e58-9ab8-04df57b6b1df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:10.087611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.077103 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2" pod "00c7e4b6-b102-4e58-9ab8-04df57b6b1df" (UID: "00c7e4b6-b102-4e58-9ab8-04df57b6b1df"). InnerVolumeSpecName "kube-api-access-glgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:10.118991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.118958 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glgh2\" (UniqueName: \"kubernetes.io/projected/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-kube-api-access-glgh2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.119222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.119208 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00c7e4b6-b102-4e58-9ab8-04df57b6b1df-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.124837 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.124809 6171 generic.go:356] "Generic (PLEG): container finished" podID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" containerID="248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa" exitCode=0 Oct 30 06:22:10.125088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.125032 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" Oct 30 06:22:10.125294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.125128 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" event={"ID":"00c7e4b6-b102-4e58-9ab8-04df57b6b1df","Type":"ContainerDied","Data":"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa"} Oct 30 06:22:10.125369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.125331 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj" event={"ID":"00c7e4b6-b102-4e58-9ab8-04df57b6b1df","Type":"ContainerDied","Data":"6b0fcf0dca0340ccb7984e909798dd9a802466f06f3d26a5d022cc9831f15239"} Oct 30 06:22:10.125369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.125366 6171 scope.go:122] "RemoveContainer" containerID="248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa" Oct 30 06:22:10.128202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.128174 6171 patch_prober.go:29] interesting pod/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:10.128308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.128227 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:10.142303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.141704 6171 generic.go:356] "Generic (PLEG): container finished" podID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerID="9bf916eb5df7e7548fa92a20ccd90bbbcaa8a1f495d14fb509758a72f77fd277" exitCode=0 Oct 30 06:22:10.142303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.141823 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" event={"ID":"f38e784f-0f80-48ac-a403-33f3254b3b4a","Type":"ContainerDied","Data":"9bf916eb5df7e7548fa92a20ccd90bbbcaa8a1f495d14fb509758a72f77fd277"} Oct 30 06:22:10.147552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.147520 6171 generic.go:356] "Generic (PLEG): container finished" podID="3dd955cd-c784-49f4-be56-f4ac59582632" containerID="6561fb6a0bf2e14303dd7191d446e5e5417a8b4e33fae51cb599e6b8fc6280dc" exitCode=0 Oct 30 06:22:10.147720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.147700 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" event={"ID":"3dd955cd-c784-49f4-be56-f4ac59582632","Type":"ContainerDied","Data":"6561fb6a0bf2e14303dd7191d446e5e5417a8b4e33fae51cb599e6b8fc6280dc"} Oct 30 06:22:10.162209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.162162 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" event={"ID":"3630cedd-62fe-4d9f-84f5-130d11293b39","Type":"ContainerDied","Data":"63f583613e79a264dd2b9e48001076a99f1f78ff87bf4ad259279725ba17f197"} Oct 30 06:22:10.162377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.161992 6171 generic.go:356] "Generic (PLEG): container finished" podID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerID="63f583613e79a264dd2b9e48001076a99f1f78ff87bf4ad259279725ba17f197" exitCode=0 Oct 30 06:22:10.164760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.164425 6171 scope.go:122] "RemoveContainer" containerID="248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa" Oct 30 06:22:10.164760 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:10.164743 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa\": container with ID starting with 248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa not found: ID does not exist" containerID="248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa" Oct 30 06:22:10.164924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.164779 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa"} err="failed to get container status \"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa\": rpc error: code = NotFound desc = could not find container \"248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa\": container with ID starting with 248bbddbcda9b61e4167423aad0e08b2d99a8192cfbe6af7b8fd1ecee4be3aaa not found: ID does not exist" Oct 30 06:22:10.167749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.167725 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:22:10.213754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.213729 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:22:10.229268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.229233 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 06:22:10.233867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.233842 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:22:10.235001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.234978 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-f22a52-895f8cfff-qblkj"] Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.282119 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58"] Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283016 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e4a02500-dd1d-49cd-a510-6b93254bafac" containerName="payload-pre-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283032 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4a02500-dd1d-49cd-a510-6b93254bafac" containerName="payload-pre-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283066 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283072 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283084 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" containerName="maas-api" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283089 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" containerName="maas-api" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283098 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d2de5d50-1858-4ed6-905b-c563cd576a57" containerName="revoke-keys" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283103 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2de5d50-1858-4ed6-905b-c563cd576a57" containerName="revoke-keys" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283115 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283120 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283137 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283145 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283153 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283157 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283177 6171 cpu_manager.go:414] "containerMap: removing container" podUID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" containerName="payload-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283181 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" containerName="payload-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283324 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" containerName="payload-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283341 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4a02500-dd1d-49cd-a510-6b93254bafac" containerName="payload-pre-processing" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283347 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283355 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8bc4bd5-c2ee-4e58-87c0-66fd3b9264a6" containerName="maas-api" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283362 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2de5d50-1858-4ed6-905b-c563cd576a57" containerName="revoke-keys" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283376 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283385 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ab1062-bade-46bb-906a-6e92a09d12f1" containerName="main" Oct 30 06:22:10.285094 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.283393 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" Oct 30 06:22:10.291084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.291030 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.304410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.304380 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58"] Oct 30 06:22:10.321264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321231 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.321424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321319 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.321424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321375 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.321424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321409 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.321600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321578 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.321659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321624 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.321713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321692 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.321800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321785 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.321863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321832 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\" (UniqueName: \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.321922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321859 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.321922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321891 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.322031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321938 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.322031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.321971 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\" (UniqueName: \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.322031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322011 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322082 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.322214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322114 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\") pod \"3dd955cd-c784-49f4-be56-f4ac59582632\" (UID: \"3dd955cd-c784-49f4-be56-f4ac59582632\") " Oct 30 06:22:10.322214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322141 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322169 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\" (UniqueName: \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.322429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322226 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\") pod \"f38e784f-0f80-48ac-a403-33f3254b3b4a\" (UID: \"f38e784f-0f80-48ac-a403-33f3254b3b4a\") " Oct 30 06:22:10.322429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322269 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.322429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322271 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322297 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322482 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322652 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.322750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.322742 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.323173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.323126 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.323527 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\" (UniqueName: \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\") pod \"3630cedd-62fe-4d9f-84f5-130d11293b39\" (UID: \"3630cedd-62fe-4d9f-84f5-130d11293b39\") " Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324740 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324760 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324770 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324783 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324795 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324805 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324816 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324826 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.324864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.324839 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.332144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.331989 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "kube-api-access-4ztf6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:10.332144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.332024 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.332478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.332415 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:10.332478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.332421 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "kube-api-access-bjlqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:10.332478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.332441 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "kube-api-access-xp99c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:10.333266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.333231 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.333525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.333478 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs" pod "3dd955cd-c784-49f4-be56-f4ac59582632" (UID: "3dd955cd-c784-49f4-be56-f4ac59582632"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:10.333647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.333534 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm" pod "3630cedd-62fe-4d9f-84f5-130d11293b39" (UID: "3630cedd-62fe-4d9f-84f5-130d11293b39"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:10.334156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.334129 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs" pod "f38e784f-0f80-48ac-a403-33f3254b3b4a" (UID: "f38e784f-0f80-48ac-a403-33f3254b3b4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:10.393962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.393922 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:22:10.394157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.394121 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:10.426278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.426502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426308 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6gm9\" (UniqueName: \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.426502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426401 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426551 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3630cedd-62fe-4d9f-84f5-130d11293b39-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426565 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f38e784f-0f80-48ac-a403-33f3254b3b4a-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426573 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3dd955cd-c784-49f4-be56-f4ac59582632-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426581 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f38e784f-0f80-48ac-a403-33f3254b3b4a-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426589 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3630cedd-62fe-4d9f-84f5-130d11293b39-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426599 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp99c\" (UniqueName: \"kubernetes.io/projected/3dd955cd-c784-49f4-be56-f4ac59582632-kube-api-access-xp99c\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426607 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3dd955cd-c784-49f4-be56-f4ac59582632-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426617 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjlqx\" (UniqueName: \"kubernetes.io/projected/f38e784f-0f80-48ac-a403-33f3254b3b4a-kube-api-access-bjlqx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.426716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.426625 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ztf6\" (UniqueName: \"kubernetes.io/projected/3630cedd-62fe-4d9f-84f5-130d11293b39-kube-api-access-4ztf6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:10.527716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.527679 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.527898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.527739 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6gm9\" (UniqueName: \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.527898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.527773 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.528656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.528582 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.530503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.530480 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.553722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.553693 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6gm9\" (UniqueName: \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.599784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.599698 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:10.611478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.611446 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:22:10.615517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.615493 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:10.618077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.618038 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.638672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.638645 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-1685dd-openshift-default-dockercfg-xjrwc\"" Oct 30 06:22:10.642877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.642855 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.730855 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.730920 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.730969 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731073 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731102 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrs4p\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731218 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731248 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731395 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.731914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.731553 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854034 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854130 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854174 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854246 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854327 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrs4p\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854458 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854562 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.854727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.854680 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.889092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.873911 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.889092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.874499 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.889092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.874709 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.889092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.874903 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.893136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.889745 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.906017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.905843 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.908233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.908096 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.916879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.916792 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrs4p\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.927354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.927319 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\") pod \"e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.942069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.941825 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:10.955390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:10.955346 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58"] Oct 30 06:22:10.957995 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:10.957951 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode81f95ea_aa38_4da1_9a62_546e4ff21931.slice/crio-ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637 WatchSource:0}: Error finding container ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637: Status 404 returned error can't find the container with id ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637 Oct 30 06:22:11.066706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.066649 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" containerName="main" probeResult="failure" output="Get \"https://10.133.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:22:11.150495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.150400 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.61:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:22:11.181896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.181805 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" Oct 30 06:22:11.182105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.181809 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w" event={"ID":"3630cedd-62fe-4d9f-84f5-130d11293b39","Type":"ContainerDied","Data":"674ee2e54c2130d5ab4c7831d9c8b0bedd33773d9721c968b84a38d4321e639c"} Oct 30 06:22:11.182105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.181977 6171 scope.go:122] "RemoveContainer" containerID="63f583613e79a264dd2b9e48001076a99f1f78ff87bf4ad259279725ba17f197" Oct 30 06:22:11.187527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.187498 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" event={"ID":"e81f95ea-aa38-4da1-9a62-546e4ff21931","Type":"ContainerStarted","Data":"ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637"} Oct 30 06:22:11.189113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.189090 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" Oct 30 06:22:11.189113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.189100 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl" event={"ID":"f38e784f-0f80-48ac-a403-33f3254b3b4a","Type":"ContainerDied","Data":"ff8d9d013dbb7b793f056c012c8702cfa866bf4549675b4b3d04192d4e6abadb"} Oct 30 06:22:11.191217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.191189 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" event={"ID":"3dd955cd-c784-49f4-be56-f4ac59582632","Type":"ContainerDied","Data":"f2407a00a99cb926bbe194acc761bfd48cd0a2fc4b2a20946937fd32f4d3076a"} Oct 30 06:22:11.191322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.191299 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx" Oct 30 06:22:11.209349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.209269 6171 scope.go:122] "RemoveContainer" containerID="9bf916eb5df7e7548fa92a20ccd90bbbcaa8a1f495d14fb509758a72f77fd277" Oct 30 06:22:11.224152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.224133 6171 scope.go:122] "RemoveContainer" containerID="6561fb6a0bf2e14303dd7191d446e5e5417a8b4e33fae51cb599e6b8fc6280dc" Oct 30 06:22:11.259725 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.259699 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 06:22:11.264334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.264308 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/distinct-w5-a35835-kserve-69cbc4dffc-c6d9w"] Oct 30 06:22:11.279270 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:11.279247 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6386b48_1c8d_4419_801d_a719daf18cc6.slice/crio-a9379260876614ff4f68fcdc0bac5df11e94cae22ab5d2da6b558b8b3ff4d59b WatchSource:0}: Error finding container a9379260876614ff4f68fcdc0bac5df11e94cae22ab5d2da6b558b8b3ff4d59b: Status 404 returned error can't find the container with id a9379260876614ff4f68fcdc0bac5df11e94cae22ab5d2da6b558b8b3ff4d59b Oct 30 06:22:11.280779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.280758 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:22:11.281972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.281941 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:11.282116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.282012 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:11.282116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.282106 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:11.320404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.320383 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 06:22:11.324607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.324586 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/simulator-w5-a35835-kserve-5c45bf874c-6vqgl"] Oct 30 06:22:11.363099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.363036 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 06:22:11.366898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.366880 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-a35835/premium-w5-a35835-kserve-647d55d48b-s6fzx"] Oct 30 06:22:11.676122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.676000 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00c7e4b6-b102-4e58-9ab8-04df57b6b1df" path="/var/lib/kubelet/pods/00c7e4b6-b102-4e58-9ab8-04df57b6b1df/volumes" Oct 30 06:22:11.676537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.676519 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3630cedd-62fe-4d9f-84f5-130d11293b39" path="/var/lib/kubelet/pods/3630cedd-62fe-4d9f-84f5-130d11293b39/volumes" Oct 30 06:22:11.676924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.676911 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd955cd-c784-49f4-be56-f4ac59582632" path="/var/lib/kubelet/pods/3dd955cd-c784-49f4-be56-f4ac59582632/volumes" Oct 30 06:22:11.677638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:11.677622 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f38e784f-0f80-48ac-a403-33f3254b3b4a" path="/var/lib/kubelet/pods/f38e784f-0f80-48ac-a403-33f3254b3b4a/volumes" Oct 30 06:22:12.203512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.203481 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerStarted","Data":"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f"} Oct 30 06:22:12.203702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.203541 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerStarted","Data":"a9379260876614ff4f68fcdc0bac5df11e94cae22ab5d2da6b558b8b3ff4d59b"} Oct 30 06:22:12.205251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.205224 6171 generic.go:356] "Generic (PLEG): container finished" podID="e81f95ea-aa38-4da1-9a62-546e4ff21931" containerID="16d12a262a0665a4adcdb049c2c5444167a7b2f60786d85d4d575c9a15a28240" exitCode=0 Oct 30 06:22:12.205376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.205300 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" event={"ID":"e81f95ea-aa38-4da1-9a62-546e4ff21931","Type":"ContainerDied","Data":"16d12a262a0665a4adcdb049c2c5444167a7b2f60786d85d4d575c9a15a28240"} Oct 30 06:22:12.284950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.284866 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podStartSLOduration=2.284851892 podStartE2EDuration="2.284851892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:12.280186029 +0000 UTC m=+2990.720563223" watchObservedRunningTime="2026-10-30 06:22:12.284851892 +0000 UTC m=+2990.725229084" Oct 30 06:22:12.763396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.763367 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45"] Oct 30 06:22:12.768120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.768100 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.800927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.800901 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45"] Oct 30 06:22:12.801092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.800961 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg"] Oct 30 06:22:12.805548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.805524 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.840097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.836788 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg"] Oct 30 06:22:12.882736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.882706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.882924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.882750 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.882924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.882870 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wtdm\" (UniqueName: \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.883196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.883175 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.883263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.883250 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.883327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.883311 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjkdk\" (UniqueName: \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.942350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.942324 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:12.947693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.947674 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:12.984542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984515 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjkdk\" (UniqueName: \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.984628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984579 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.984628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984603 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.984722 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984654 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wtdm\" (UniqueName: \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.984812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984797 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.984881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.984840 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.985389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.985367 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:12.985493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.985423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.987361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.987343 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:12.987442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:12.987423 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:13.010323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.010296 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjkdk\" (UniqueName: \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\") pod \"maas-api-revoke-sub-c761b88d-6v9xg\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:13.010423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.010330 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wtdm\" (UniqueName: \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\") pod \"maas-api-revoke-sub-9155c284-wwq45\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:13.097124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.089440 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:13.131236 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.129345 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:13.223347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.223134 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:13.223347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.223232 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:22:13.631036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.630972 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:13.696742 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:13.696691 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce5ffeae_7635_47ad_a73e_f168befb6d39.slice/crio-aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555 WatchSource:0}: Error finding container aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555: Status 404 returned error can't find the container with id aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555 Oct 30 06:22:13.701276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.701256 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk"] Oct 30 06:22:13.703125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.703104 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45"] Oct 30 06:22:13.722373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.722351 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-197e9b-qpbhk"] Oct 30 06:22:13.724085 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:13.724032 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb42d5c9_d109_478a_b05b_5a8f037ebdea.slice/crio-5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e WatchSource:0}: Error finding container 5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e: Status 404 returned error can't find the container with id 5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e Oct 30 06:22:13.748855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.741374 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg"] Oct 30 06:22:13.748855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.759772 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\" (UniqueName: \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\") pod \"e81f95ea-aa38-4da1-9a62-546e4ff21931\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " Oct 30 06:22:13.787717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.787671 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9" pod "e81f95ea-aa38-4da1-9a62-546e4ff21931" (UID: "e81f95ea-aa38-4da1-9a62-546e4ff21931"). InnerVolumeSpecName "kube-api-access-f6gm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:13.790656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.788880 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\") pod \"e81f95ea-aa38-4da1-9a62-546e4ff21931\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " Oct 30 06:22:13.790656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.788978 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\") pod \"e81f95ea-aa38-4da1-9a62-546e4ff21931\" (UID: \"e81f95ea-aa38-4da1-9a62-546e4ff21931\") " Oct 30 06:22:13.791543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.791498 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca" pod "e81f95ea-aa38-4da1-9a62-546e4ff21931" (UID: "e81f95ea-aa38-4da1-9a62-546e4ff21931"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:13.792738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.792714 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca" pod "e81f95ea-aa38-4da1-9a62-546e4ff21931" (UID: "e81f95ea-aa38-4da1-9a62-546e4ff21931"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:13.793226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.793207 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.793309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.793228 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6gm9\" (UniqueName: \"kubernetes.io/projected/e81f95ea-aa38-4da1-9a62-546e4ff21931-kube-api-access-f6gm9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:13.793309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:13.793239 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e81f95ea-aa38-4da1-9a62-546e4ff21931-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:14.231368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.231343 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" Oct 30 06:22:14.231490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.231399 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58" event={"ID":"e81f95ea-aa38-4da1-9a62-546e4ff21931","Type":"ContainerDied","Data":"ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637"} Oct 30 06:22:14.231490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.231442 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba63cb753f58699e0673e027bdfe90fff8374223b8e0d3eeaee3d76209f3e637" Oct 30 06:22:14.233331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.233299 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" event={"ID":"ce5ffeae-7635-47ad-a73e-f168befb6d39","Type":"ContainerStarted","Data":"801d3e5ad7e59356876dcd203e0b6185bbd753a083950151d5cd1c469f4dd23c"} Oct 30 06:22:14.233458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.233355 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" event={"ID":"ce5ffeae-7635-47ad-a73e-f168befb6d39","Type":"ContainerStarted","Data":"aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555"} Oct 30 06:22:14.235218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.235188 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" event={"ID":"cb42d5c9-d109-478a-b05b-5a8f037ebdea","Type":"ContainerStarted","Data":"5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e"} Oct 30 06:22:14.279831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.279753 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" podStartSLOduration=2.279739685 podStartE2EDuration="2.279739685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:14.27703882 +0000 UTC m=+2992.717416040" watchObservedRunningTime="2026-10-30 06:22:14.279739685 +0000 UTC m=+2992.720116878" Oct 30 06:22:14.725537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.725508 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:14.726358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.726340 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e81f95ea-aa38-4da1-9a62-546e4ff21931" containerName="revoke-keys" Oct 30 06:22:14.726459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.726360 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e81f95ea-aa38-4da1-9a62-546e4ff21931" containerName="revoke-keys" Oct 30 06:22:14.726583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.726572 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e81f95ea-aa38-4da1-9a62-546e4ff21931" containerName="revoke-keys" Oct 30 06:22:14.730554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.730533 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.753380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.753354 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:14.806758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.806729 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.806867 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.806978 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807019 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807069 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fndn4\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807124 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807164 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807236 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.807567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.807277 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.903520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.903483 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:22:14.903785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.903758 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" containerID="cri-o://3c4551783472c27a0885433564a0544a72797e1c4797f3d7639061b89690d9e3" gracePeriod=30 Oct 30 06:22:14.909472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.909329 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.909737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.909705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911123 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fndn4\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911214 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911286 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911373 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911418 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911473 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.911627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.911611 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.912393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.912364 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.912663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.912640 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.913779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.913164 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.913779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.913417 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.913779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.913742 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.914329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.914300 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.916760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.916714 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.943896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.943873 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:14.960307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:14.960218 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fndn4\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\") pod \"e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:15.088008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.087756 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:15.289690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.289397 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:15.290102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.289971 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:15.388156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.384499 6171 generic.go:356] "Generic (PLEG): container finished" podID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerID="9bd63cfed53ee35b54a37c48bbbc2bd9e5ce5363b699b1839aceec09aaacc010" exitCode=0 Oct 30 06:22:15.388156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.384655 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" event={"ID":"fd11f58b-7756-452e-be90-d1e8b2b3dafd","Type":"ContainerDied","Data":"9bd63cfed53ee35b54a37c48bbbc2bd9e5ce5363b699b1839aceec09aaacc010"} Oct 30 06:22:15.389342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.388667 6171 generic.go:356] "Generic (PLEG): container finished" podID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerID="8fa59c1d018244ae21cc2e4bcc6a804f8effd34700ae432b3f2ca965b808e37d" exitCode=0 Oct 30 06:22:15.389342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.388788 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" event={"ID":"172e7059-b3b9-4a9c-9441-1f2905b4ec3d","Type":"ContainerDied","Data":"8fa59c1d018244ae21cc2e4bcc6a804f8effd34700ae432b3f2ca965b808e37d"} Oct 30 06:22:15.394601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.393683 6171 generic.go:356] "Generic (PLEG): container finished" podID="ce5ffeae-7635-47ad-a73e-f168befb6d39" containerID="801d3e5ad7e59356876dcd203e0b6185bbd753a083950151d5cd1c469f4dd23c" exitCode=0 Oct 30 06:22:15.394601 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.393809 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" event={"ID":"ce5ffeae-7635-47ad-a73e-f168befb6d39","Type":"ContainerDied","Data":"801d3e5ad7e59356876dcd203e0b6185bbd753a083950151d5cd1c469f4dd23c"} Oct 30 06:22:15.417937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.417731 6171 generic.go:356] "Generic (PLEG): container finished" podID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerID="d8bc8392161b62ae3f512013c0100fd97b527c42c7c4df16d4098c26381d8392" exitCode=0 Oct 30 06:22:15.417937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.417875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" event={"ID":"c7244299-ce2c-4c21-bd1b-fded0b5c174f","Type":"ContainerDied","Data":"d8bc8392161b62ae3f512013c0100fd97b527c42c7c4df16d4098c26381d8392"} Oct 30 06:22:15.422271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.420966 6171 generic.go:356] "Generic (PLEG): container finished" podID="cb42d5c9-d109-478a-b05b-5a8f037ebdea" containerID="c9b59038b1b72a35af1272f382ad0a409c72399e34aa25ccb20c434c059498ed" exitCode=0 Oct 30 06:22:15.422271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.421033 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" event={"ID":"cb42d5c9-d109-478a-b05b-5a8f037ebdea","Type":"ContainerDied","Data":"c9b59038b1b72a35af1272f382ad0a409c72399e34aa25ccb20c434c059498ed"} Oct 30 06:22:15.425655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.425632 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5"] Oct 30 06:22:15.433405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.433385 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.456179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.456089 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5"] Oct 30 06:22:15.471474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.471211 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:22:15.486367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.486277 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:22:15.497254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.495708 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.497254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.495886 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.497254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.495922 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwcj\" (UniqueName: \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.519394 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.516653 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:22:15.522691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.522665 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5"] Oct 30 06:22:15.524128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524106 6171 cpu_manager.go:414] "containerMap: removing container" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" Oct 30 06:22:15.524271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524259 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" Oct 30 06:22:15.524386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524377 6171 cpu_manager.go:414] "containerMap: removing container" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" Oct 30 06:22:15.524471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524459 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" Oct 30 06:22:15.524590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524580 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" Oct 30 06:22:15.524663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524654 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" Oct 30 06:22:15.524955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.524944 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" containerName="istio-proxy" Oct 30 06:22:15.525082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.525072 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" containerName="istio-proxy" Oct 30 06:22:15.525178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.525170 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" containerName="istio-proxy" Oct 30 06:22:15.534755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.534732 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.544432 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.544409 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5"] Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597559 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597627 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597671 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597803 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597837 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597868 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597911 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597940 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.597977 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598089 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598144 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598196 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598241 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598271 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598312 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598345 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598395 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598444 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598502 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\") pod \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\" (UID: \"c7244299-ce2c-4c21-bd1b-fded0b5c174f\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598532 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598582 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598637 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598690 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598756 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\") pod \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\" (UID: \"172e7059-b3b9-4a9c-9441-1f2905b4ec3d\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598826 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598890 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.598932 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\") pod \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\" (UID: \"fd11f58b-7756-452e-be90-d1e8b2b3dafd\") " Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599233 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwqw\" (UniqueName: \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599414 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599462 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599503 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwcj\" (UniqueName: \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599712 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.599946 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.600637 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.600786 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.600881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.601159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.600908 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.601261 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.603472 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.603663 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.604476 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "kube-api-access-lcrmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.604550 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.604784 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.605433 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.605561 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.605741 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.605882 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606025 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.605499 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "kube-api-access-ncm45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606411 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606460 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606604 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606788 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "kube-api-access-s7ml7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606874 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.606923 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.607721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.607174 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.609612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.609523 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.610762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.610734 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy" pod "c7244299-ce2c-4c21-bd1b-fded0b5c174f" (UID: "c7244299-ce2c-4c21-bd1b-fded0b5c174f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:15.610863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.610782 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token" pod "172e7059-b3b9-4a9c-9441-1f2905b4ec3d" (UID: "172e7059-b3b9-4a9c-9441-1f2905b4ec3d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.611876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.611818 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:15.611876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.611846 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token" pod "fd11f58b-7756-452e-be90-d1e8b2b3dafd" (UID: "fd11f58b-7756-452e-be90-d1e8b2b3dafd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.633015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.632982 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwcj\" (UniqueName: \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.670088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.667929 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:15.670088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.668017 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:15.670088 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.668075 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:22:15.723076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.723020 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.723470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.723447 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwqw\" (UniqueName: \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.723714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.723697 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.724020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724002 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724169 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724263 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7ml7\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-kube-api-access-s7ml7\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724357 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724452 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724547 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724638 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724779 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724861 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.724970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.724960 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725064 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725168 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcrmx\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-kube-api-access-lcrmx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725257 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725348 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725436 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725517 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725603 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725696 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725788 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/172e7059-b3b9-4a9c-9441-1f2905b4ec3d-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725878 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd11f58b-7756-452e-be90-d1e8b2b3dafd-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.725981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.725970 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726093 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd11f58b-7756-452e-be90-d1e8b2b3dafd-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726210 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726312 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726397 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7244299-ce2c-4c21-bd1b-fded0b5c174f-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726485 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7244299-ce2c-4c21-bd1b-fded0b5c174f-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.726584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.726573 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ncm45\" (UniqueName: \"kubernetes.io/projected/c7244299-ce2c-4c21-bd1b-fded0b5c174f-kube-api-access-ncm45\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.727353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.727330 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.727920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.727897 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:22:15.741583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.740805 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.744224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.744198 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8722a33-9b7a-439f-9693-a05cea761700" path="/var/lib/kubelet/pods/e8722a33-9b7a-439f-9693-a05cea761700/volumes" Oct 30 06:22:15.745345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.745325 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:15.758957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.758935 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:22:15.759160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.759081 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwqw\" (UniqueName: \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\") pod \"maas-api-revoke-sub-6a043a72-fk9j5\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.813959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.813928 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:15.828139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.828111 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\" (UniqueName: \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\") pod \"93a08c43-2493-47c8-9501-ed90af6f9972\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " Oct 30 06:22:15.828258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.828173 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\") pod \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " Oct 30 06:22:15.828484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.828459 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\" (UniqueName: \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\") pod \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\" (UID: \"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6\") " Oct 30 06:22:15.828643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.828613 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume" pod "44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" (UID: "44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.828928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.828905 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\") pod \"93a08c43-2493-47c8-9501-ed90af6f9972\" (UID: \"93a08c43-2493-47c8-9501-ed90af6f9972\") " Oct 30 06:22:15.829797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.829765 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.830300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.830272 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume" pod "93a08c43-2493-47c8-9501-ed90af6f9972" (UID: "93a08c43-2493-47c8-9501-ed90af6f9972"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:15.831235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.831215 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n" pod "44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" (UID: "44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6"). InnerVolumeSpecName "kube-api-access-wnb2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.831991 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.831967 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q" pod "93a08c43-2493-47c8-9501-ed90af6f9972" (UID: "93a08c43-2493-47c8-9501-ed90af6f9972"). InnerVolumeSpecName "kube-api-access-g728q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:15.867062 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.866804 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:15.933605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.933577 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnb2n\" (UniqueName: \"kubernetes.io/projected/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6-kube-api-access-wnb2n\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.933605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.933608 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93a08c43-2493-47c8-9501-ed90af6f9972-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:15.933768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:15.933621 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g728q\" (UniqueName: \"kubernetes.io/projected/93a08c43-2493-47c8-9501-ed90af6f9972-kube-api-access-g728q\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:16.155903 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:16.155856 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177e7ab8_58db_4b64_8078_2cd669c8b1e4.slice/crio-960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f WatchSource:0}: Error finding container 960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f: Status 404 returned error can't find the container with id 960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f Oct 30 06:22:16.158989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.158752 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5"] Oct 30 06:22:16.263813 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:16.263765 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9651c604_5fb3_4494_b6d2_05f4794ce49e.slice/crio-dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd WatchSource:0}: Error finding container dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd: Status 404 returned error can't find the container with id dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd Oct 30 06:22:16.264769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.264669 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5"] Oct 30 06:22:16.442794 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.442761 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" event={"ID":"19f5fcf1-a619-494a-bf4f-ce031051a4f0","Type":"ContainerStarted","Data":"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588"} Oct 30 06:22:16.443006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.442822 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" event={"ID":"19f5fcf1-a619-494a-bf4f-ce031051a4f0","Type":"ContainerStarted","Data":"c91e1e6115a58fdee526cd77b3d19b77b0097a033ddaca0cb49ae3b6a813be57"} Oct 30 06:22:16.444528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.444503 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" Oct 30 06:22:16.444528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.444521 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb" event={"ID":"fd11f58b-7756-452e-be90-d1e8b2b3dafd","Type":"ContainerDied","Data":"42b064007b16f2e58f60ba785ca29adec3351cab4459a03812eff2bfff131df1"} Oct 30 06:22:16.444724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.444571 6171 scope.go:122] "RemoveContainer" containerID="9bd63cfed53ee35b54a37c48bbbc2bd9e5ce5363b699b1839aceec09aaacc010" Oct 30 06:22:16.446237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.446215 6171 generic.go:356] "Generic (PLEG): container finished" podID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerID="c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581" exitCode=0 Oct 30 06:22:16.446357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.446258 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" Oct 30 06:22:16.446357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.446302 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" event={"ID":"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6","Type":"ContainerDied","Data":"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581"} Oct 30 06:22:16.446357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.446347 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9" event={"ID":"44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6","Type":"ContainerDied","Data":"1f7ea6297e7272de346229af62aee1f0d264685de69818c7b6341a28267f9b80"} Oct 30 06:22:16.448350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.448323 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" Oct 30 06:22:16.448443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.448335 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq" event={"ID":"172e7059-b3b9-4a9c-9441-1f2905b4ec3d","Type":"ContainerDied","Data":"061d4fa5c9335a8bfd7a3f6fe2a0dacd0c7ecf14e11ce577862cc4ae41e01cf5"} Oct 30 06:22:16.450180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.450157 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" event={"ID":"177e7ab8-58db-4b64-8078-2cd669c8b1e4","Type":"ContainerStarted","Data":"960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f"} Oct 30 06:22:16.452356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.452326 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" Oct 30 06:22:16.452476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.452368 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc" event={"ID":"c7244299-ce2c-4c21-bd1b-fded0b5c174f","Type":"ContainerDied","Data":"49c9c51e92bacd35d02daa581b154f36e627b500a771a928c217bd225524eb62"} Oct 30 06:22:16.454267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.454246 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" event={"ID":"9651c604-5fb3-4494-b6d2-05f4794ce49e","Type":"ContainerStarted","Data":"dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd"} Oct 30 06:22:16.456138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.456113 6171 generic.go:356] "Generic (PLEG): container finished" podID="93a08c43-2493-47c8-9501-ed90af6f9972" containerID="fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246" exitCode=0 Oct 30 06:22:16.456608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.456323 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" event={"ID":"93a08c43-2493-47c8-9501-ed90af6f9972","Type":"ContainerDied","Data":"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246"} Oct 30 06:22:16.456608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.456548 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" event={"ID":"93a08c43-2493-47c8-9501-ed90af6f9972","Type":"ContainerDied","Data":"91b00b3919ccb61adff36070d2a675533c47c7e4c70061c7e07f7356d57e1e8a"} Oct 30 06:22:16.456608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.456354 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r" Oct 30 06:22:16.461605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.461589 6171 scope.go:122] "RemoveContainer" containerID="c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581" Oct 30 06:22:16.474347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.474329 6171 scope.go:122] "RemoveContainer" containerID="c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581" Oct 30 06:22:16.474629 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:16.474607 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581\": container with ID starting with c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581 not found: ID does not exist" containerID="c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581" Oct 30 06:22:16.474701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.474635 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581"} err="failed to get container status \"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581\": rpc error: code = NotFound desc = could not find container \"c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581\": container with ID starting with c9a9f6033def9c0543c8c1f1607ce0ccc6ac0448be31c022e7e507f5e7af6581 not found: ID does not exist" Oct 30 06:22:16.474701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.474649 6171 scope.go:122] "RemoveContainer" containerID="8fa59c1d018244ae21cc2e4bcc6a804f8effd34700ae432b3f2ca965b808e37d" Oct 30 06:22:16.527612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.527482 6171 scope.go:122] "RemoveContainer" containerID="d8bc8392161b62ae3f512013c0100fd97b527c42c7c4df16d4098c26381d8392" Oct 30 06:22:16.544231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.544204 6171 scope.go:122] "RemoveContainer" containerID="fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246" Oct 30 06:22:16.564318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.564292 6171 scope.go:122] "RemoveContainer" containerID="fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246" Oct 30 06:22:16.565154 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:16.565122 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246\": container with ID starting with fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246 not found: ID does not exist" containerID="fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246" Oct 30 06:22:16.565328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.565299 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246"} err="failed to get container status \"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246\": rpc error: code = NotFound desc = could not find container \"fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246\": container with ID starting with fd669f99be9e87c66d56c3dd37719fb79b08259aa73ca4e03add044454cea246 not found: ID does not exist" Oct 30 06:22:16.572500 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.572439 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podStartSLOduration=2.572419472 podStartE2EDuration="2.572419472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:16.522417582 +0000 UTC m=+2994.962794775" watchObservedRunningTime="2026-10-30 06:22:16.572419472 +0000 UTC m=+2995.012796665" Oct 30 06:22:16.576712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.576626 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:22:16.620067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.620016 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-kbspb"] Oct 30 06:22:16.645899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.645861 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:22:16.654087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.654057 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-v5dkq"] Oct 30 06:22:16.724280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.724244 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:22:16.731061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.731019 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-24db22-698b4cf795-crxz9"] Oct 30 06:22:16.774744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.774703 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:22:16.822690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.822508 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-24db22-68d6bd569f-4zw7r"] Oct 30 06:22:16.832784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.832725 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:22:16.838326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.838291 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-5deb36-openshift-default-5c58465fc5-mdddc"] Oct 30 06:22:16.844224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.844200 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:16.851602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.850775 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\") pod \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " Oct 30 06:22:16.851602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.850898 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\") pod \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " Oct 30 06:22:16.851602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.850964 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\" (UniqueName: \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\") pod \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\" (UID: \"cb42d5c9-d109-478a-b05b-5a8f037ebdea\") " Oct 30 06:22:16.851969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.851910 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca" pod "cb42d5c9-d109-478a-b05b-5a8f037ebdea" (UID: "cb42d5c9-d109-478a-b05b-5a8f037ebdea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:16.852862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.852845 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:16.853422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.853394 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk" pod "cb42d5c9-d109-478a-b05b-5a8f037ebdea" (UID: "cb42d5c9-d109-478a-b05b-5a8f037ebdea"). InnerVolumeSpecName "kube-api-access-cjkdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:16.853610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.853589 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca" pod "cb42d5c9-d109-478a-b05b-5a8f037ebdea" (UID: "cb42d5c9-d109-478a-b05b-5a8f037ebdea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.972935 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\") pod \"ce5ffeae-7635-47ad-a73e-f168befb6d39\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.973195 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\") pod \"ce5ffeae-7635-47ad-a73e-f168befb6d39\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.973253 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\" (UniqueName: \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\") pod \"ce5ffeae-7635-47ad-a73e-f168befb6d39\" (UID: \"ce5ffeae-7635-47ad-a73e-f168befb6d39\") " Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.973848 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.973868 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb42d5c9-d109-478a-b05b-5a8f037ebdea-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.973882 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjkdk\" (UniqueName: \"kubernetes.io/projected/cb42d5c9-d109-478a-b05b-5a8f037ebdea-kube-api-access-cjkdk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:16.981107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:16.976744 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca" pod "ce5ffeae-7635-47ad-a73e-f168befb6d39" (UID: "ce5ffeae-7635-47ad-a73e-f168befb6d39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:17.098118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.054201 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca" pod "ce5ffeae-7635-47ad-a73e-f168befb6d39" (UID: "ce5ffeae-7635-47ad-a73e-f168befb6d39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:17.195956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.139648 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:17.195956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.146256 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm" pod "ce5ffeae-7635-47ad-a73e-f168befb6d39" (UID: "ce5ffeae-7635-47ad-a73e-f168befb6d39"). InnerVolumeSpecName "kube-api-access-8wtdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:17.195956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.160299 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:17.195956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.160364 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:17.196668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.196596 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:17.196668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.196628 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce5ffeae-7635-47ad-a73e-f168befb6d39-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:17.196668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.196645 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wtdm\" (UniqueName: \"kubernetes.io/projected/ce5ffeae-7635-47ad-a73e-f168befb6d39-kube-api-access-8wtdm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:17.230229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.230181 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:22:17.231415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231395 6171 cpu_manager.go:414] "containerMap: removing container" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" Oct 30 06:22:17.231503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231418 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" Oct 30 06:22:17.231503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231436 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ffeae-7635-47ad-a73e-f168befb6d39" containerName="revoke-keys" Oct 30 06:22:17.231503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231443 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ffeae-7635-47ad-a73e-f168befb6d39" containerName="revoke-keys" Oct 30 06:22:17.231503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231487 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cb42d5c9-d109-478a-b05b-5a8f037ebdea" containerName="revoke-keys" Oct 30 06:22:17.231503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231495 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb42d5c9-d109-478a-b05b-5a8f037ebdea" containerName="revoke-keys" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231508 6171 cpu_manager.go:414] "containerMap: removing container" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231515 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231699 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb42d5c9-d109-478a-b05b-5a8f037ebdea" containerName="revoke-keys" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231711 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" containerName="payload-pre-processing" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231730 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" containerName="payload-processing" Oct 30 06:22:17.231757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.231741 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ffeae-7635-47ad-a73e-f168befb6d39" containerName="revoke-keys" Oct 30 06:22:17.238458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.238433 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:22:17.240423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.239295 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.256668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.256627 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1685dd-dockercfg-dhn7z\"" Oct 30 06:22:17.256779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.256706 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1685dd\"" Oct 30 06:22:17.302640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.302139 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stgc9\" (UniqueName: \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.302640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.302237 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.406531 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.406231 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.406697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.406541 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stgc9\" (UniqueName: \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.407627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.407606 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.432038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.432013 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stgc9\" (UniqueName: \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\") pod \"payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.475664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.475632 6171 generic.go:356] "Generic (PLEG): container finished" podID="177e7ab8-58db-4b64-8078-2cd669c8b1e4" containerID="d998517744e841002c9e4b3afabab7ac1bb31c6ed0d355600e26314dd4a0962c" exitCode=0 Oct 30 06:22:17.475806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.475712 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" event={"ID":"177e7ab8-58db-4b64-8078-2cd669c8b1e4","Type":"ContainerDied","Data":"d998517744e841002c9e4b3afabab7ac1bb31c6ed0d355600e26314dd4a0962c"} Oct 30 06:22:17.477207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.477188 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" Oct 30 06:22:17.477324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.477229 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45" event={"ID":"ce5ffeae-7635-47ad-a73e-f168befb6d39","Type":"ContainerDied","Data":"aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555"} Oct 30 06:22:17.477324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.477271 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aecc6dc9f54271d7e4478f2ee43cd6232aebdb84c23debeaf6f92fb1936f8555" Oct 30 06:22:17.480004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.479986 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" Oct 30 06:22:17.480152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.480020 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg" event={"ID":"cb42d5c9-d109-478a-b05b-5a8f037ebdea","Type":"ContainerDied","Data":"5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e"} Oct 30 06:22:17.501074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.493162 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5169804d10ec6dcab43012fe07e87bb383700d40b7be5184f6f0c1ab515f8b1e" Oct 30 06:22:17.508588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.508555 6171 generic.go:356] "Generic (PLEG): container finished" podID="9651c604-5fb3-4494-b6d2-05f4794ce49e" containerID="92b443ba67ac0909d01349c0e913b2314cc9b6c3587802c824c12bf1c7f66c2c" exitCode=0 Oct 30 06:22:17.508744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.508650 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" event={"ID":"9651c604-5fb3-4494-b6d2-05f4794ce49e","Type":"ContainerDied","Data":"92b443ba67ac0909d01349c0e913b2314cc9b6c3587802c824c12bf1c7f66c2c"} Oct 30 06:22:17.560783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.560754 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:17.732875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.732833 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="172e7059-b3b9-4a9c-9441-1f2905b4ec3d" path="/var/lib/kubelet/pods/172e7059-b3b9-4a9c-9441-1f2905b4ec3d/volumes" Oct 30 06:22:17.733519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.733499 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6" path="/var/lib/kubelet/pods/44a32ecf-a6f2-4cbf-97b5-24866ecf0fe6/volumes" Oct 30 06:22:17.734093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.734075 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93a08c43-2493-47c8-9501-ed90af6f9972" path="/var/lib/kubelet/pods/93a08c43-2493-47c8-9501-ed90af6f9972/volumes" Oct 30 06:22:17.735254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.735235 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7244299-ce2c-4c21-bd1b-fded0b5c174f" path="/var/lib/kubelet/pods/c7244299-ce2c-4c21-bd1b-fded0b5c174f/volumes" Oct 30 06:22:17.735804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.735781 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd11f58b-7756-452e-be90-d1e8b2b3dafd" path="/var/lib/kubelet/pods/fd11f58b-7756-452e-be90-d1e8b2b3dafd/volumes" Oct 30 06:22:17.911152 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:17.911123 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded11bf34_35ff_45f5_aee1_d262a9541428.slice/crio-6aa768d678a7668a2d7b3b1320739a8daa39f98cb15d63d77c6482de6552fa6c WatchSource:0}: Error finding container 6aa768d678a7668a2d7b3b1320739a8daa39f98cb15d63d77c6482de6552fa6c: Status 404 returned error can't find the container with id 6aa768d678a7668a2d7b3b1320739a8daa39f98cb15d63d77c6482de6552fa6c Oct 30 06:22:17.911494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:17.911387 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:22:18.103070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.102519 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:18.103070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.102583 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:18.521267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.521221 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" event={"ID":"ed11bf34-35ff-45f5-aee1-d262a9541428","Type":"ContainerStarted","Data":"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324"} Oct 30 06:22:18.521469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.521289 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" event={"ID":"ed11bf34-35ff-45f5-aee1-d262a9541428","Type":"ContainerStarted","Data":"6aa768d678a7668a2d7b3b1320739a8daa39f98cb15d63d77c6482de6552fa6c"} Oct 30 06:22:18.521469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.521319 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:18.565605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.565556 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" podStartSLOduration=2.565543741 podStartE2EDuration="2.565543741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:18.56530285 +0000 UTC m=+2997.005680043" watchObservedRunningTime="2026-10-30 06:22:18.565543741 +0000 UTC m=+2997.005920934" Oct 30 06:22:18.827389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.827359 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:18.865006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.864981 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942228 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\") pod \"9651c604-5fb3-4494-b6d2-05f4794ce49e\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942371 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\" (UniqueName: \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\") pod \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942432 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\") pod \"9651c604-5fb3-4494-b6d2-05f4794ce49e\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942474 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\") pod \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942550 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\" (UniqueName: \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\") pod \"9651c604-5fb3-4494-b6d2-05f4794ce49e\" (UID: \"9651c604-5fb3-4494-b6d2-05f4794ce49e\") " Oct 30 06:22:18.943077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.942582 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\") pod \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\" (UID: \"177e7ab8-58db-4b64-8078-2cd669c8b1e4\") " Oct 30 06:22:18.944446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.943092 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca" pod "9651c604-5fb3-4494-b6d2-05f4794ce49e" (UID: "9651c604-5fb3-4494-b6d2-05f4794ce49e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:18.944446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.943385 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:18.944446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.943420 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca" pod "177e7ab8-58db-4b64-8078-2cd669c8b1e4" (UID: "177e7ab8-58db-4b64-8078-2cd669c8b1e4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:18.946423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.946398 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw" pod "9651c604-5fb3-4494-b6d2-05f4794ce49e" (UID: "9651c604-5fb3-4494-b6d2-05f4794ce49e"). InnerVolumeSpecName "kube-api-access-dlwqw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:18.947123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.947096 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca" pod "177e7ab8-58db-4b64-8078-2cd669c8b1e4" (UID: "177e7ab8-58db-4b64-8078-2cd669c8b1e4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:18.947240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.947224 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca" pod "9651c604-5fb3-4494-b6d2-05f4794ce49e" (UID: "9651c604-5fb3-4494-b6d2-05f4794ce49e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:18.948225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:18.948200 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj" pod "177e7ab8-58db-4b64-8078-2cd669c8b1e4" (UID: "177e7ab8-58db-4b64-8078-2cd669c8b1e4"). InnerVolumeSpecName "kube-api-access-6xwcj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:19.044524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.044496 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwqw\" (UniqueName: \"kubernetes.io/projected/9651c604-5fb3-4494-b6d2-05f4794ce49e-kube-api-access-dlwqw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.044524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.044524 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.044726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.044538 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9651c604-5fb3-4494-b6d2-05f4794ce49e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.044726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.044551 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xwcj\" (UniqueName: \"kubernetes.io/projected/177e7ab8-58db-4b64-8078-2cd669c8b1e4-kube-api-access-6xwcj\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.044726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.044565 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/177e7ab8-58db-4b64-8078-2cd669c8b1e4-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:19.106235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.105667 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:19.106235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.105751 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:19.533890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.533854 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" Oct 30 06:22:19.534110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.533894 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5" event={"ID":"9651c604-5fb3-4494-b6d2-05f4794ce49e","Type":"ContainerDied","Data":"dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd"} Oct 30 06:22:19.534110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.533948 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc42cff4e2c6872a28e7c75ec5d88d4a8f056b72c843fb644fe8d19872250dcd" Oct 30 06:22:19.535700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.535669 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" Oct 30 06:22:19.535700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.535692 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5" event={"ID":"177e7ab8-58db-4b64-8078-2cd669c8b1e4","Type":"ContainerDied","Data":"960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f"} Oct 30 06:22:19.535940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:19.535737 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="960fd05ed08057b05f4738e9143b7c3ec470736928dd3bfefb0d71a1bfcdc70f" Oct 30 06:22:20.106081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.103272 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:20.106081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.103341 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:20.549210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.549172 6171 generic.go:356] "Generic (PLEG): container finished" podID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerID="3c4551783472c27a0885433564a0544a72797e1c4797f3d7639061b89690d9e3" exitCode=0 Oct 30 06:22:20.549422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.549237 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerDied","Data":"3c4551783472c27a0885433564a0544a72797e1c4797f3d7639061b89690d9e3"} Oct 30 06:22:20.549422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.549303 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" event={"ID":"e39d80a0-0fb5-49f5-a77a-b5e54e700e62","Type":"ContainerDied","Data":"779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75"} Oct 30 06:22:20.549422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.549323 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="779a792511bd159ca37d6f88104167b31358d0a154161eed92e9fa299a2c7f75" Oct 30 06:22:20.549422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.549339 6171 scope.go:122] "RemoveContainer" containerID="939405f21115328c1b3a1001de04c79b481a9a8c1474879382f2e5606dd07805" Oct 30 06:22:20.553118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.553082 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:22:20.723631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723589 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.723852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723657 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.723852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723695 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.723852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723811 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723868 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723921 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723942 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.724092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.723971 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724037 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724154 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.724430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724191 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\") pod \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\" (UID: \"e39d80a0-0fb5-49f5-a77a-b5e54e700e62\") " Oct 30 06:22:20.724430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724331 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:20.724430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724388 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.724430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.724415 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.725286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.725261 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.725286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.725290 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.725493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.725305 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.725493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.725320 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.725493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.725334 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.726462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.726432 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:20.726797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.726777 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "kube-api-access-svkjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:20.727297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.727271 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:20.727393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.727303 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy" pod "e39d80a0-0fb5-49f5-a77a-b5e54e700e62" (UID: "e39d80a0-0fb5-49f5-a77a-b5e54e700e62"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:20.832806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.832323 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 06:22:20.838484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.834710 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.838484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.834759 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svkjh\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-kube-api-access-svkjh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.838484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.834774 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.838484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.834789 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e39d80a0-0fb5-49f5-a77a-b5e54e700e62-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.850186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:20.850139 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 06:22:21.106128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.105437 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:21.106128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.105499 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:21.131660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.131623 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 06:22:21.131904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.131876 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" containerID="cri-o://abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89" gracePeriod=30 Oct 30 06:22:21.564661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.564633 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l" Oct 30 06:22:21.789082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.775738 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:22:21.834109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:21.832029 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-24db22-openshift-default-879d78fb5-4cj7l"] Oct 30 06:22:22.092155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:22.089420 6171 patch_prober.go:29] interesting pod/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" start-of-body= Oct 30 06:22:22.092155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:22.089480 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.160:15021/healthz/ready\": dial tcp 10.133.1.160:15021: connect: connection refused" Oct 30 06:22:23.094503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:23.094474 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:23.094984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:23.094966 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:23.095729 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:23.095709 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:23.674231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:23.674194 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" path="/var/lib/kubelet/pods/e39d80a0-0fb5-49f5-a77a-b5e54e700e62/volumes" Oct 30 06:22:25.177186 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:25.177166 6171 logging.go:55] [core] [Channel #17908 SubChannel #17910]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.53:50052", ServerName: "10.133.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.53:50052: connect: connection refused" Oct 30 06:22:25.177817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.177792 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:25.507719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.507688 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 06:22:25.531637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.531611 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 06:22:25.591001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.590968 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\" (UniqueName: \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\") pod \"114634d3-0554-4801-8c34-866fc41d1666\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " Oct 30 06:22:25.591234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.591075 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\") pod \"114634d3-0554-4801-8c34-866fc41d1666\" (UID: \"114634d3-0554-4801-8c34-866fc41d1666\") " Oct 30 06:22:25.591234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.591108 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\" (UniqueName: \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\") pod \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " Oct 30 06:22:25.591376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.591355 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\") pod \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\" (UID: \"4d5001f6-5cd5-43de-aa4c-d47ddd912639\") " Oct 30 06:22:25.591522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.591497 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume" pod "114634d3-0554-4801-8c34-866fc41d1666" (UID: "114634d3-0554-4801-8c34-866fc41d1666"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:25.591872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.591844 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume" pod "4d5001f6-5cd5-43de-aa4c-d47ddd912639" (UID: "4d5001f6-5cd5-43de-aa4c-d47ddd912639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:25.592077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.592056 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d5001f6-5cd5-43de-aa4c-d47ddd912639-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.592159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.592083 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/114634d3-0554-4801-8c34-866fc41d1666-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.594083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.594036 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd" pod "114634d3-0554-4801-8c34-866fc41d1666" (UID: "114634d3-0554-4801-8c34-866fc41d1666"). InnerVolumeSpecName "kube-api-access-hnnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:25.594178 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.594097 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq" pod "4d5001f6-5cd5-43de-aa4c-d47ddd912639" (UID: "4d5001f6-5cd5-43de-aa4c-d47ddd912639"). InnerVolumeSpecName "kube-api-access-vw2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:25.611824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.611784 6171 generic.go:356] "Generic (PLEG): container finished" podID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerID="be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4" exitCode=0 Oct 30 06:22:25.612011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.611845 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" Oct 30 06:22:25.612011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.611957 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" event={"ID":"4d5001f6-5cd5-43de-aa4c-d47ddd912639","Type":"ContainerDied","Data":"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4"} Oct 30 06:22:25.612011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.612005 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt" event={"ID":"4d5001f6-5cd5-43de-aa4c-d47ddd912639","Type":"ContainerDied","Data":"18239f2cc51f0910b272de5fb98a149197949b9e2bf3a322f3ae83e8e8608913"} Oct 30 06:22:25.612269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.612033 6171 scope.go:122] "RemoveContainer" containerID="be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4" Oct 30 06:22:25.614119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.614088 6171 generic.go:356] "Generic (PLEG): container finished" podID="114634d3-0554-4801-8c34-866fc41d1666" containerID="0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c" exitCode=0 Oct 30 06:22:25.614248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.614222 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" event={"ID":"114634d3-0554-4801-8c34-866fc41d1666","Type":"ContainerDied","Data":"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c"} Oct 30 06:22:25.614370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.614274 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" event={"ID":"114634d3-0554-4801-8c34-866fc41d1666","Type":"ContainerDied","Data":"b3f45ec5d3817f822bdc36fff77c63df53f40ee2c4fc6975dfbf658d3dd5a7ab"} Oct 30 06:22:25.614370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.614343 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" Oct 30 06:22:25.628557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.628539 6171 scope.go:122] "RemoveContainer" containerID="be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4" Oct 30 06:22:25.628862 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:25.628839 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4\": container with ID starting with be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4 not found: ID does not exist" containerID="be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4" Oct 30 06:22:25.628969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.628870 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4"} err="failed to get container status \"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4\": rpc error: code = NotFound desc = could not find container \"be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4\": container with ID starting with be61113979f48fb0890bdbd0b200e51440dd9546a257c6ce6e34872fb5a144b4 not found: ID does not exist" Oct 30 06:22:25.628969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.628896 6171 scope.go:122] "RemoveContainer" containerID="0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c" Oct 30 06:22:25.641073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.641014 6171 scope.go:122] "RemoveContainer" containerID="0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c" Oct 30 06:22:25.641453 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:25.641427 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c\": container with ID starting with 0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c not found: ID does not exist" containerID="0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c" Oct 30 06:22:25.641539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.641464 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c"} err="failed to get container status \"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c\": rpc error: code = NotFound desc = could not find container \"0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c\": container with ID starting with 0e1e76bc86561d7b2a9c6bdb744379f87a680efbcb97e7bec0fc7027190cb51c not found: ID does not exist" Oct 30 06:22:25.676659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.676593 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 06:22:25.677826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.677804 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-a35835-577d4c7ddc-nn5pt"] Oct 30 06:22:25.693038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.693012 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnnvd\" (UniqueName: \"kubernetes.io/projected/114634d3-0554-4801-8c34-866fc41d1666-kube-api-access-hnnvd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.693038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.693038 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw2kq\" (UniqueName: \"kubernetes.io/projected/4d5001f6-5cd5-43de-aa4c-d47ddd912639-kube-api-access-vw2kq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.713560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.713540 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 06:22:25.718278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.718253 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h"] Oct 30 06:22:25.849070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.846575 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" containerID="cri-o://076af070dea7b7308f62f09d9e3073313d82989c5fa521f236fc3f409dcf7e67" gracePeriod=25 Oct 30 06:22:25.862350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.857229 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" containerID="cri-o://19d6310029f68204e47c79901b5b69257fee36b768594346e685cbfe5e1292ec" gracePeriod=25 Oct 30 06:22:25.873938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.873905 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:22:25.878748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:25.878719 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:22:26.177141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.177107 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-a35835-9765b9fcf-gqg7h" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.53:50052\" within 1s: context deadline exceeded" Oct 30 06:22:26.393490 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:26.393454 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:22:26.410061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.409983 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 06:22:26.514443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514399 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514487 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514557 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514606 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514698 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514801 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.514884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514830 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.515116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.514967 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.515116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.515024 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\") pod \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\" (UID: \"e2f5320e-e28b-46eb-b2d8-6f63e83f2274\") " Oct 30 06:22:26.530737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.528459 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.530737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.528662 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.530737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.528936 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:26.530737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.530686 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.531114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.530810 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.577324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.577258 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:26.622732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622695 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.622732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622734 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.622732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622749 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.623128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622765 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.623128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622781 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.623128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.622793 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.642961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.639402 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:26.645280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.645250 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "kube-api-access-fbhfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:26.651071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.650637 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy" pod "e2f5320e-e28b-46eb-b2d8-6f63e83f2274" (UID: "e2f5320e-e28b-46eb-b2d8-6f63e83f2274"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:26.663380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.661553 6171 generic.go:356] "Generic (PLEG): container finished" podID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerID="abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89" exitCode=0 Oct 30 06:22:26.663380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.661674 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" Oct 30 06:22:26.663380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.663094 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" event={"ID":"e2f5320e-e28b-46eb-b2d8-6f63e83f2274","Type":"ContainerDied","Data":"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89"} Oct 30 06:22:26.663380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.663243 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv" event={"ID":"e2f5320e-e28b-46eb-b2d8-6f63e83f2274","Type":"ContainerDied","Data":"c2d5f790aa35bb961562042cc5b3577db48f8845450e3437f2e922ce56248c29"} Oct 30 06:22:26.664146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.664120 6171 scope.go:122] "RemoveContainer" containerID="abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89" Oct 30 06:22:26.682516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.682490 6171 scope.go:122] "RemoveContainer" containerID="abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89" Oct 30 06:22:26.682813 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:26.682791 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89\": container with ID starting with abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89 not found: ID does not exist" containerID="abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89" Oct 30 06:22:26.683496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.682826 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89"} err="failed to get container status \"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89\": rpc error: code = NotFound desc = could not find container \"abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89\": container with ID starting with abb13b149481b39a057841f59df4d703e23727ed00d7c99bcca559ee87b28d89 not found: ID does not exist" Oct 30 06:22:26.724981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.724944 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.724981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.724982 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.724981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.724997 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbhfv\" (UniqueName: \"kubernetes.io/projected/e2f5320e-e28b-46eb-b2d8-6f63e83f2274-kube-api-access-fbhfv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:26.726175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.726149 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 06:22:26.730294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:26.730271 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-197e9b-openshift-default-5784796cff-lr8dv"] Oct 30 06:22:27.674269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:27.674238 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="114634d3-0554-4801-8c34-866fc41d1666" path="/var/lib/kubelet/pods/114634d3-0554-4801-8c34-866fc41d1666/volumes" Oct 30 06:22:27.674733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:27.674670 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" path="/var/lib/kubelet/pods/4d5001f6-5cd5-43de-aa4c-d47ddd912639/volumes" Oct 30 06:22:27.675122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:27.675107 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" path="/var/lib/kubelet/pods/e2f5320e-e28b-46eb-b2d8-6f63e83f2274/volumes" Oct 30 06:22:28.569547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:28.569514 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:28.569547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:28.569529 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:29.562070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:29.561650 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:22:29.966928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:29.966835 6171 scope.go:122] "RemoveContainer" containerID="c8f44ec3c1b840ff1220d3af550352ffd2e158438aca23d679f44b4ef47ee8ba" Oct 30 06:22:29.979731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:29.979703 6171 scope.go:122] "RemoveContainer" containerID="218f508c80fa86a433adc0a7fd3a590ee4d15e8618446e0a1fd2f475e9652e40" Oct 30 06:22:29.993211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:29.993191 6171 scope.go:122] "RemoveContainer" containerID="27acc75bdde1b3084ef57dd1fd66b0001b45984954a6d7d1e80e5c6e950d8fa3" Oct 30 06:22:30.003863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:30.003837 6171 scope.go:122] "RemoveContainer" containerID="3c4551783472c27a0885433564a0544a72797e1c4797f3d7639061b89690d9e3" Oct 30 06:22:30.874926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:30.874880 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" containerID="cri-o://d5fe23e5b4cb0cc28197a73097b170ee2fef39d81d4939936557a87201edb495" gracePeriod=25 Oct 30 06:22:30.879450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:30.879411 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" containerID="cri-o://2d4f261b5342d60be847d5d6140f2ea8d81243aa1f5bc5492c6c4771e2591464" gracePeriod=25 Oct 30 06:22:32.531068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:32.531022 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc"] Oct 30 06:22:32.544129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:32.544103 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f22a52-45jdc"] Oct 30 06:22:33.674214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:33.674177 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="822d3556-d424-438e-a4c8-54282e213241" path="/var/lib/kubelet/pods/822d3556-d424-438e-a4c8-54282e213241/volumes" Oct 30 06:22:36.028693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:36.028652 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:36.029182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:36.028698 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:38.618070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:38.617615 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:38.618070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:38.617692 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:38.945380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:38.945304 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5"] Oct 30 06:22:38.962105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:38.962072 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-0f980e-4sph5"] Oct 30 06:22:39.490590 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:39.490545 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58"] Oct 30 06:22:39.504469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:39.504442 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-6b5b1e-d7v58"] Oct 30 06:22:39.694124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:39.692845 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177e7ab8-58db-4b64-8078-2cd669c8b1e4" path="/var/lib/kubelet/pods/177e7ab8-58db-4b64-8078-2cd669c8b1e4/volumes" Oct 30 06:22:39.694124 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:39.693609 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e81f95ea-aa38-4da1-9a62-546e4ff21931" path="/var/lib/kubelet/pods/e81f95ea-aa38-4da1-9a62-546e4ff21931/volumes" Oct 30 06:22:40.117919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.117873 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:22:40.118682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118657 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" Oct 30 06:22:40.118682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118682 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118710 6171 cpu_manager.go:414] "containerMap: removing container" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118717 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118735 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9651c604-5fb3-4494-b6d2-05f4794ce49e" containerName="revoke-keys" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118742 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9651c604-5fb3-4494-b6d2-05f4794ce49e" containerName="revoke-keys" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118770 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118777 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118795 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118802 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118817 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118824 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118842 6171 cpu_manager.go:414] "containerMap: removing container" podUID="177e7ab8-58db-4b64-8078-2cd669c8b1e4" containerName="revoke-keys" Oct 30 06:22:40.118878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.118849 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177e7ab8-58db-4b64-8078-2cd669c8b1e4" containerName="revoke-keys" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119065 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d5001f6-5cd5-43de-aa4c-d47ddd912639" containerName="payload-pre-processing" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119083 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="114634d3-0554-4801-8c34-866fc41d1666" containerName="payload-processing" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119098 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9651c604-5fb3-4494-b6d2-05f4794ce49e" containerName="revoke-keys" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119112 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119123 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f5320e-e28b-46eb-b2d8-6f63e83f2274" containerName="istio-proxy" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119133 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39d80a0-0fb5-49f5-a77a-b5e54e700e62" containerName="istio-proxy" Oct 30 06:22:40.119515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.119144 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="177e7ab8-58db-4b64-8078-2cd669c8b1e4" containerName="revoke-keys" Oct 30 06:22:40.126755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.126734 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.144313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.144293 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:22:40.152360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.152331 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-1685dd-serving-cert\"" Oct 30 06:22:40.199278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.199250 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf"] Oct 30 06:22:40.208271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.208238 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-a35835-gd9hf"] Oct 30 06:22:40.219420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.219390 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.219602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.219579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.219689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.219675 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.219766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.219710 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7sx6\" (UniqueName: \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.321431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.321385 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.321609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.321569 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.321674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.321653 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.321733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.321699 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7sx6\" (UniqueName: \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.322571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.322542 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.331317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.331285 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.359431 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.359348 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7sx6\" (UniqueName: \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.360332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.360170 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\") pod \"maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.442447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.442360 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:40.798267 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:40.798235 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed23fda_b84c_449c_8b28_3a73be116291.slice/crio-82f821f84022ccb3e4a4f4579cc18bbdac9516560d21754c31147fa0548f49a9 WatchSource:0}: Error finding container 82f821f84022ccb3e4a4f4579cc18bbdac9516560d21754c31147fa0548f49a9: Status 404 returned error can't find the container with id 82f821f84022ccb3e4a4f4579cc18bbdac9516560d21754c31147fa0548f49a9 Oct 30 06:22:40.798625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.798508 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:22:40.799971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.799956 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:22:40.819332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:40.819304 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" event={"ID":"fed23fda-b84c-449c-8b28-3a73be116291","Type":"ContainerStarted","Data":"82f821f84022ccb3e4a4f4579cc18bbdac9516560d21754c31147fa0548f49a9"} Oct 30 06:22:41.707279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:41.707238 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74b5a263-88ca-46e2-9898-dc12e3139af0" path="/var/lib/kubelet/pods/74b5a263-88ca-46e2-9898-dc12e3139af0/volumes" Oct 30 06:22:41.707998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:41.707973 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 06:22:41.708306 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:41.708222 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" containerID="cri-o://3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c" gracePeriod=30 Oct 30 06:22:41.909255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:41.909093 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" event={"ID":"fed23fda-b84c-449c-8b28-3a73be116291","Type":"ContainerStarted","Data":"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da"} Oct 30 06:22:41.923413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:41.910330 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:42.404806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.404755 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" podStartSLOduration=2.017450276 podStartE2EDuration="2.404739463s" totalImagesPullingTime="387.289187ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:41.95893871 +0000 UTC m=+3020.399315905" watchObservedRunningTime="2026-10-30 06:22:42.404739463 +0000 UTC m=+3020.845116656" Oct 30 06:22:42.408941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.408916 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s"] Oct 30 06:22:42.414355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.414336 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.432682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.432652 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:22:42.432808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.432734 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:42.437098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.437069 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s"] Oct 30 06:22:42.466004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.465978 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.466260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.466230 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbsx\" (UniqueName: \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.466369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.466275 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.567563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.567534 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.567708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.567678 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbsx\" (UniqueName: \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.567751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.567711 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.568397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.568295 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.570311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.570279 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.594242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.594217 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbsx\" (UniqueName: \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\") pod \"maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:42.730654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:42.730625 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:43.057800 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:43.057762 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e558d9_d6c2_490a_a3de_9384a45a7e34.slice/crio-947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059 WatchSource:0}: Error finding container 947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059: Status 404 returned error can't find the container with id 947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059 Oct 30 06:22:43.058287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:43.058263 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s"] Oct 30 06:22:43.927860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:43.927783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" event={"ID":"67e558d9-d6c2-490a-a3de-9384a45a7e34","Type":"ContainerStarted","Data":"cb951db38b26a7dcdbd6af51b554ed0821e5aa1e98f5fce8dca9069e60be8f6f"} Oct 30 06:22:43.927860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:43.927837 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" event={"ID":"67e558d9-d6c2-490a-a3de-9384a45a7e34","Type":"ContainerStarted","Data":"947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059"} Oct 30 06:22:43.975604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:43.975541 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" podStartSLOduration=1.975528685 podStartE2EDuration="1.975528685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:43.972260891 +0000 UTC m=+3022.412638084" watchObservedRunningTime="2026-10-30 06:22:43.975528685 +0000 UTC m=+3022.415905877" Oct 30 06:22:44.016845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.016817 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:22:44.017074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.017015 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" podUID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" containerName="maas-api" containerID="cri-o://6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f" gracePeriod=30 Oct 30 06:22:44.187605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.187535 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:22:44.188070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.187758 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" podUID="9d014002-9bed-4379-aea2-89b267686ae5" containerName="payload-processing" containerID="cri-o://1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea" gracePeriod=30 Oct 30 06:22:44.212077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.212014 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:22:44.212298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.212274 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" podUID="5e47b155-107e-4652-a357-52cea322229e" containerName="payload-pre-processing" containerID="cri-o://ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b" gracePeriod=30 Oct 30 06:22:44.713065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.712982 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:22:44.732888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.732856 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\") pod \"9d014002-9bed-4379-aea2-89b267686ae5\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " Oct 30 06:22:44.733097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.733034 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\" (UniqueName: \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\") pod \"9d014002-9bed-4379-aea2-89b267686ae5\" (UID: \"9d014002-9bed-4379-aea2-89b267686ae5\") " Oct 30 06:22:44.733271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.733249 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume" pod "9d014002-9bed-4379-aea2-89b267686ae5" (UID: "9d014002-9bed-4379-aea2-89b267686ae5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.733566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.733547 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d014002-9bed-4379-aea2-89b267686ae5-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.735206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.735185 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5" pod "9d014002-9bed-4379-aea2-89b267686ae5" (UID: "9d014002-9bed-4379-aea2-89b267686ae5"). InnerVolumeSpecName "kube-api-access-nwjw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.766902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.766880 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:22:44.770262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.770242 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:22:44.835692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835661 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\" (UniqueName: \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\") pod \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " Oct 30 06:22:44.835864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835724 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") pod \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " Oct 30 06:22:44.835864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835785 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\") pod \"5e47b155-107e-4652-a357-52cea322229e\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " Oct 30 06:22:44.835864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835853 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\" (UniqueName: \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\") pod \"5e47b155-107e-4652-a357-52cea322229e\" (UID: \"5e47b155-107e-4652-a357-52cea322229e\") " Oct 30 06:22:44.836109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835911 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\") pod \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " Oct 30 06:22:44.836109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.835951 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\") pod \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\" (UID: \"6143e0ba-e5c5-41d5-9937-53e7a706cd55\") " Oct 30 06:22:44.836586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.836557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume" pod "5e47b155-107e-4652-a357-52cea322229e" (UID: "5e47b155-107e-4652-a357-52cea322229e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.836673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.836618 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca" pod "6143e0ba-e5c5-41d5-9937-53e7a706cd55" (UID: "6143e0ba-e5c5-41d5-9937-53e7a706cd55"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:44.837208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.837184 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e47b155-107e-4652-a357-52cea322229e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.837315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.837212 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjw5\" (UniqueName: \"kubernetes.io/projected/9d014002-9bed-4379-aea2-89b267686ae5-kube-api-access-nwjw5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.837315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.837228 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6143e0ba-e5c5-41d5-9937-53e7a706cd55-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.839586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.839562 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr" pod "5e47b155-107e-4652-a357-52cea322229e" (UID: "5e47b155-107e-4652-a357-52cea322229e"). InnerVolumeSpecName "kube-api-access-6hmkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.840166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.840142 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg" pod "6143e0ba-e5c5-41d5-9937-53e7a706cd55" (UID: "6143e0ba-e5c5-41d5-9937-53e7a706cd55"). InnerVolumeSpecName "kube-api-access-m6hbg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:44.840474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.840446 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs" pod "6143e0ba-e5c5-41d5-9937-53e7a706cd55" (UID: "6143e0ba-e5c5-41d5-9937-53e7a706cd55"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:44.840693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.840675 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls" pod "6143e0ba-e5c5-41d5-9937-53e7a706cd55" (UID: "6143e0ba-e5c5-41d5-9937-53e7a706cd55"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:44.842818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.842790 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs"] Oct 30 06:22:44.848488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.848464 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-bf87e1-mf6fs"] Oct 30 06:22:44.949081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.945772 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6hmkr\" (UniqueName: \"kubernetes.io/projected/5e47b155-107e-4652-a357-52cea322229e-kube-api-access-6hmkr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.949081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.945813 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.949081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.945828 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6hbg\" (UniqueName: \"kubernetes.io/projected/6143e0ba-e5c5-41d5-9937-53e7a706cd55-kube-api-access-m6hbg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.949081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.945866 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6143e0ba-e5c5-41d5-9937-53e7a706cd55-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:44.982834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.982796 6171 generic.go:356] "Generic (PLEG): container finished" podID="5e47b155-107e-4652-a357-52cea322229e" containerID="ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b" exitCode=0 Oct 30 06:22:44.983073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.982847 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" Oct 30 06:22:44.983073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.982854 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" event={"ID":"5e47b155-107e-4652-a357-52cea322229e","Type":"ContainerDied","Data":"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b"} Oct 30 06:22:44.983073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.982956 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57" event={"ID":"5e47b155-107e-4652-a357-52cea322229e","Type":"ContainerDied","Data":"070282c2ad5f80df439bb8db701f0625360b6ea42b9c73e6a49bf930176268f6"} Oct 30 06:22:44.983073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.982989 6171 scope.go:122] "RemoveContainer" containerID="ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b" Oct 30 06:22:44.984776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.984750 6171 generic.go:356] "Generic (PLEG): container finished" podID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" containerID="6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f" exitCode=0 Oct 30 06:22:44.985036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.984785 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" event={"ID":"6143e0ba-e5c5-41d5-9937-53e7a706cd55","Type":"ContainerDied","Data":"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f"} Oct 30 06:22:44.985209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.985130 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" event={"ID":"6143e0ba-e5c5-41d5-9937-53e7a706cd55","Type":"ContainerDied","Data":"5ad5b91a75ac9746d65b2035562bb2b4200eed61c285409245bd63a9795644eb"} Oct 30 06:22:44.985209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.984842 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz" Oct 30 06:22:44.987007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.986985 6171 generic.go:356] "Generic (PLEG): container finished" podID="67e558d9-d6c2-490a-a3de-9384a45a7e34" containerID="cb951db38b26a7dcdbd6af51b554ed0821e5aa1e98f5fce8dca9069e60be8f6f" exitCode=0 Oct 30 06:22:44.987202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.987095 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" event={"ID":"67e558d9-d6c2-490a-a3de-9384a45a7e34","Type":"ContainerDied","Data":"cb951db38b26a7dcdbd6af51b554ed0821e5aa1e98f5fce8dca9069e60be8f6f"} Oct 30 06:22:44.988796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.988773 6171 generic.go:356] "Generic (PLEG): container finished" podID="9d014002-9bed-4379-aea2-89b267686ae5" containerID="1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea" exitCode=0 Oct 30 06:22:44.988925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.988805 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" Oct 30 06:22:44.988925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.988845 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" event={"ID":"9d014002-9bed-4379-aea2-89b267686ae5","Type":"ContainerDied","Data":"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea"} Oct 30 06:22:44.988925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:44.988886 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn" event={"ID":"9d014002-9bed-4379-aea2-89b267686ae5","Type":"ContainerDied","Data":"575f56ba757df7a876ae7e552e84e976126a22f3deb1362ec8a76df116fa72fd"} Oct 30 06:22:45.004463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.004444 6171 scope.go:122] "RemoveContainer" containerID="ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b" Oct 30 06:22:45.004756 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:45.004733 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b\": container with ID starting with ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b not found: ID does not exist" containerID="ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b" Oct 30 06:22:45.004849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.004764 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b"} err="failed to get container status \"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b\": rpc error: code = NotFound desc = could not find container \"ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b\": container with ID starting with ac133585c7fd1479148091d51204c9fc9d8e0ee97f9fde611e181d3d6041847b not found: ID does not exist" Oct 30 06:22:45.004849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.004792 6171 scope.go:122] "RemoveContainer" containerID="6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f" Oct 30 06:22:45.016356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.016327 6171 scope.go:122] "RemoveContainer" containerID="6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f" Oct 30 06:22:45.016617 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:45.016597 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f\": container with ID starting with 6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f not found: ID does not exist" containerID="6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f" Oct 30 06:22:45.016694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.016628 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f"} err="failed to get container status \"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f\": rpc error: code = NotFound desc = could not find container \"6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f\": container with ID starting with 6ef4cefee79273bd1cdbe53b56ebd545da879771f47d19c7a9463f3fe103966f not found: ID does not exist" Oct 30 06:22:45.016694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.016651 6171 scope.go:122] "RemoveContainer" containerID="1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea" Oct 30 06:22:45.027834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.027820 6171 scope.go:122] "RemoveContainer" containerID="1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea" Oct 30 06:22:45.028098 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:45.028067 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea\": container with ID starting with 1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea not found: ID does not exist" containerID="1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea" Oct 30 06:22:45.028203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.028104 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea"} err="failed to get container status \"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea\": rpc error: code = NotFound desc = could not find container \"1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea\": container with ID starting with 1a7fe57ffc773293b08c222d02434cf5115ec96657a7c81a289c42a07b8ed0ea not found: ID does not exist" Oct 30 06:22:45.110222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.110191 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:22:45.114755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.114728 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-bf87e1-7f5f4f7cb7-pkl57"] Oct 30 06:22:45.169539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.169509 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:22:45.173629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.173607 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-bf87e1-6774bdcbcd-44wrz"] Oct 30 06:22:45.211446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.211420 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:22:45.214965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.214945 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-bf87e1-5d55c5689b-c8dqn"] Oct 30 06:22:45.673568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.673538 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e47b155-107e-4652-a357-52cea322229e" path="/var/lib/kubelet/pods/5e47b155-107e-4652-a357-52cea322229e/volumes" Oct 30 06:22:45.674021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.673992 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" path="/var/lib/kubelet/pods/6143e0ba-e5c5-41d5-9937-53e7a706cd55/volumes" Oct 30 06:22:45.674499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.674479 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d014002-9bed-4379-aea2-89b267686ae5" path="/var/lib/kubelet/pods/9d014002-9bed-4379-aea2-89b267686ae5/volumes" Oct 30 06:22:45.675317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:45.675296 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2de5d50-1858-4ed6-905b-c563cd576a57" path="/var/lib/kubelet/pods/d2de5d50-1858-4ed6-905b-c563cd576a57/volumes" Oct 30 06:22:46.010890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.010852 6171 generic.go:356] "Generic (PLEG): container finished" podID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerID="076af070dea7b7308f62f09d9e3073313d82989c5fa521f236fc3f409dcf7e67" exitCode=0 Oct 30 06:22:46.011241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.010950 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" event={"ID":"12401bb7-cf57-45fd-b716-0f8b020e48ba","Type":"ContainerDied","Data":"076af070dea7b7308f62f09d9e3073313d82989c5fa521f236fc3f409dcf7e67"} Oct 30 06:22:46.015923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.015878 6171 generic.go:356] "Generic (PLEG): container finished" podID="4f943a43-895b-4347-ae66-93b478d336e2" containerID="19d6310029f68204e47c79901b5b69257fee36b768594346e685cbfe5e1292ec" exitCode=0 Oct 30 06:22:46.016141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.015958 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" event={"ID":"4f943a43-895b-4347-ae66-93b478d336e2","Type":"ContainerDied","Data":"19d6310029f68204e47c79901b5b69257fee36b768594346e685cbfe5e1292ec"} Oct 30 06:22:46.029827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.029744 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:46.030714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.030665 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:46.169966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.169934 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 06:22:46.194275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.194210 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 06:22:46.223430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.223407 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.265693 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\" (UniqueName: \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\") pod \"67e558d9-d6c2-490a-a3de-9384a45a7e34\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.265785 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\") pod \"67e558d9-d6c2-490a-a3de-9384a45a7e34\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.267659 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\") pod \"4f943a43-895b-4347-ae66-93b478d336e2\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.267705 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\") pod \"67e558d9-d6c2-490a-a3de-9384a45a7e34\" (UID: \"67e558d9-d6c2-490a-a3de-9384a45a7e34\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.267797 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\") pod \"12401bb7-cf57-45fd-b716-0f8b020e48ba\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.267956 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\" (UniqueName: \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\") pod \"12401bb7-cf57-45fd-b716-0f8b020e48ba\" (UID: \"12401bb7-cf57-45fd-b716-0f8b020e48ba\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.267997 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\" (UniqueName: \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\") pod \"4f943a43-895b-4347-ae66-93b478d336e2\" (UID: \"4f943a43-895b-4347-ae66-93b478d336e2\") " Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.268732 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca" pod "67e558d9-d6c2-490a-a3de-9384a45a7e34" (UID: "67e558d9-d6c2-490a-a3de-9384a45a7e34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.269131 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume" pod "4f943a43-895b-4347-ae66-93b478d336e2" (UID: "4f943a43-895b-4347-ae66-93b478d336e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.269238 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca" pod "67e558d9-d6c2-490a-a3de-9384a45a7e34" (UID: "67e558d9-d6c2-490a-a3de-9384a45a7e34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.269584 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx" pod "67e558d9-d6c2-490a-a3de-9384a45a7e34" (UID: "67e558d9-d6c2-490a-a3de-9384a45a7e34"). InnerVolumeSpecName "kube-api-access-vvbsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:46.269700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.269609 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume" pod "12401bb7-cf57-45fd-b716-0f8b020e48ba" (UID: "12401bb7-cf57-45fd-b716-0f8b020e48ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:46.271386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.271353 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp" pod "12401bb7-cf57-45fd-b716-0f8b020e48ba" (UID: "12401bb7-cf57-45fd-b716-0f8b020e48ba"). InnerVolumeSpecName "kube-api-access-kgwzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:46.273003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.272979 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg" pod "4f943a43-895b-4347-ae66-93b478d336e2" (UID: "4f943a43-895b-4347-ae66-93b478d336e2"). InnerVolumeSpecName "kube-api-access-ggqrg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:46.370263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370235 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgwzp\" (UniqueName: \"kubernetes.io/projected/12401bb7-cf57-45fd-b716-0f8b020e48ba-kube-api-access-kgwzp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370261 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggqrg\" (UniqueName: \"kubernetes.io/projected/4f943a43-895b-4347-ae66-93b478d336e2-kube-api-access-ggqrg\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370270 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvbsx\" (UniqueName: \"kubernetes.io/projected/67e558d9-d6c2-490a-a3de-9384a45a7e34-kube-api-access-vvbsx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370279 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370288 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f943a43-895b-4347-ae66-93b478d336e2-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370297 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67e558d9-d6c2-490a-a3de-9384a45a7e34-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:46.370512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:46.370306 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12401bb7-cf57-45fd-b716-0f8b020e48ba-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.028271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.028229 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" Oct 30 06:22:47.028271 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.028244 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt" event={"ID":"12401bb7-cf57-45fd-b716-0f8b020e48ba","Type":"ContainerDied","Data":"3dee8de708f084a045570d2dc191fb6c91544513e01052f97baf7aede3123e02"} Oct 30 06:22:47.028556 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.028306 6171 scope.go:122] "RemoveContainer" containerID="076af070dea7b7308f62f09d9e3073313d82989c5fa521f236fc3f409dcf7e67" Oct 30 06:22:47.029911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.029889 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" Oct 30 06:22:47.030002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.029892 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms" event={"ID":"4f943a43-895b-4347-ae66-93b478d336e2","Type":"ContainerDied","Data":"2d38c34022875f9b03b14346ee98abc1bc490ec3eb65ac7b50600874aabf4146"} Oct 30 06:22:47.031146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.031124 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 06:22:47.031447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.031431 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" Oct 30 06:22:47.031549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.031480 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s" event={"ID":"67e558d9-d6c2-490a-a3de-9384a45a7e34","Type":"ContainerDied","Data":"947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059"} Oct 30 06:22:47.031549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.031524 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947ed7667cefa49d4c23c6fe4a43e8d491a8214184ccac97999d930264472059" Oct 30 06:22:47.033608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.033582 6171 generic.go:356] "Generic (PLEG): container finished" podID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerID="3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c" exitCode=0 Oct 30 06:22:47.033735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.033707 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerDied","Data":"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c"} Oct 30 06:22:47.033802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.033736 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" event={"ID":"7ce686bc-222c-4f3d-bfae-1862a05db9b7","Type":"ContainerDied","Data":"04b41a8803918073c126e423f1b343d2759d52393920a916f1408045550bbef8"} Oct 30 06:22:47.033802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.033789 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv" Oct 30 06:22:47.042622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.042607 6171 scope.go:122] "RemoveContainer" containerID="19d6310029f68204e47c79901b5b69257fee36b768594346e685cbfe5e1292ec" Oct 30 06:22:47.053453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.053438 6171 scope.go:122] "RemoveContainer" containerID="3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c" Oct 30 06:22:47.079099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.077711 6171 scope.go:122] "RemoveContainer" containerID="358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d" Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.079780 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.079842 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.079931 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080097 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080135 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080213 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080260 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080379 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.080504 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\") pod \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\" (UID: \"7ce686bc-222c-4f3d-bfae-1862a05db9b7\") " Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.081654 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.082331 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.082464 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.082877 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:47.085469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.083644 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:47.087788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.087709 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:47.087894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.087779 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:47.088016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.087991 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "kube-api-access-n5tcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:47.089471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.089443 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo" pod "7ce686bc-222c-4f3d-bfae-1862a05db9b7" (UID: "7ce686bc-222c-4f3d-bfae-1862a05db9b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:47.107363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.107340 6171 scope.go:122] "RemoveContainer" containerID="3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c" Oct 30 06:22:47.107754 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:47.107728 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c\": container with ID starting with 3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c not found: ID does not exist" containerID="3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c" Oct 30 06:22:47.107853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.107764 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c"} err="failed to get container status \"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c\": rpc error: code = NotFound desc = could not find container \"3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c\": container with ID starting with 3ff12c5f4ceaad94c4fadb2b8fed3a8539da286d70c8de20e89f035e40abce8c not found: ID does not exist" Oct 30 06:22:47.107853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.107782 6171 scope.go:122] "RemoveContainer" containerID="358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d" Oct 30 06:22:47.108100 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:47.108075 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d\": container with ID starting with 358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d not found: ID does not exist" containerID="358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d" Oct 30 06:22:47.108165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.108111 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d"} err="failed to get container status \"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d\": rpc error: code = NotFound desc = could not find container \"358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d\": container with ID starting with 358de87529500748b74d873912b045d295e73d4bc4af99dae2aa01e609a7935d not found: ID does not exist" Oct 30 06:22:47.111983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.111964 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 06:22:47.126204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.126176 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6b5b1e-7d4dc45c65-bz4gt"] Oct 30 06:22:47.170332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.170307 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 06:22:47.174735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.174717 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-6b5b1e-6b5b887499-dkpms"] Oct 30 06:22:47.178691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.178671 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 06:22:47.182421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182400 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182422 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182430 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182440 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182448 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5tcc\" (UniqueName: \"kubernetes.io/projected/7ce686bc-222c-4f3d-bfae-1862a05db9b7-kube-api-access-n5tcc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182457 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182465 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182473 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.182608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.182480 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ce686bc-222c-4f3d-bfae-1862a05db9b7-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.380348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.380309 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:47.380730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.380539 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" containerID="cri-o://5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588" gracePeriod=30 Oct 30 06:22:47.393228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.393198 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 06:22:47.397075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.397029 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-f22a52-openshift-default-8c5d8496f-vjtzv"] Oct 30 06:22:47.401433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.401413 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:22:47.401600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.401582 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" containerID="cri-o://e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f" gracePeriod=30 Oct 30 06:22:47.674177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.674099 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" path="/var/lib/kubelet/pods/12401bb7-cf57-45fd-b716-0f8b020e48ba/volumes" Oct 30 06:22:47.674588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.674573 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f943a43-895b-4347-ae66-93b478d336e2" path="/var/lib/kubelet/pods/4f943a43-895b-4347-ae66-93b478d336e2/volumes" Oct 30 06:22:47.675029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:47.675012 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" path="/var/lib/kubelet/pods/7ce686bc-222c-4f3d-bfae-1862a05db9b7/volumes" Oct 30 06:22:48.069105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069077 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7"] Oct 30 06:22:48.069736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069724 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069739 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069750 6171 cpu_manager.go:414] "containerMap: removing container" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069756 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069768 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069773 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069783 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" containerName="maas-api" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069788 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" containerName="maas-api" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069798 6171 cpu_manager.go:414] "containerMap: removing container" podUID="67e558d9-d6c2-490a-a3de-9384a45a7e34" containerName="revoke-keys" Oct 30 06:22:48.069800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069802 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67e558d9-d6c2-490a-a3de-9384a45a7e34" containerName="revoke-keys" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069811 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5e47b155-107e-4652-a357-52cea322229e" containerName="payload-pre-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069816 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e47b155-107e-4652-a357-52cea322229e" containerName="payload-pre-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069831 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069836 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069846 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9d014002-9bed-4379-aea2-89b267686ae5" containerName="payload-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069850 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d014002-9bed-4379-aea2-89b267686ae5" containerName="payload-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069951 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069958 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="12401bb7-cf57-45fd-b716-0f8b020e48ba" containerName="payload-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069966 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f943a43-895b-4347-ae66-93b478d336e2" containerName="payload-pre-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069975 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6143e0ba-e5c5-41d5-9937-53e7a706cd55" containerName="maas-api" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069982 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="67e558d9-d6c2-490a-a3de-9384a45a7e34" containerName="revoke-keys" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069989 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e47b155-107e-4652-a357-52cea322229e" containerName="payload-pre-processing" Oct 30 06:22:48.070129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.069997 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d014002-9bed-4379-aea2-89b267686ae5" containerName="payload-processing" Oct 30 06:22:48.075586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.075565 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.092831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.092795 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:48.092980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.092945 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:22:48.093501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.093478 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7"] Oct 30 06:22:48.200073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.200019 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5h8j\" (UniqueName: \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.200239 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.200134 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.200284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.200257 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.304000 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.303948 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.304263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.304224 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5h8j\" (UniqueName: \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.304355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.304291 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.305120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.305022 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.334848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.334767 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.347754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.347711 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5h8j\" (UniqueName: \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\") pod \"maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.401083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.394603 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:48.730681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.730652 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7"] Oct 30 06:22:48.731424 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:48.731398 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54be536c_6fe9_45e4_9057_38bdc0e9d8a1.slice/crio-2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d WatchSource:0}: Error finding container 2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d: Status 404 returned error can't find the container with id 2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d Oct 30 06:22:48.757376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.757353 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2"] Oct 30 06:22:48.758205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.758190 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce686bc-222c-4f3d-bfae-1862a05db9b7" containerName="istio-proxy" Oct 30 06:22:48.763286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.763265 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.782935 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.782912 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2"] Oct 30 06:22:48.812127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.812097 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24fcf\" (UniqueName: \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.812295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.812209 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.812295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.812256 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.913327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.913294 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.913486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.913445 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24fcf\" (UniqueName: \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.913574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.913553 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.913992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.913972 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.915955 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.915935 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:48.954103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.949967 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:22:48.972123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:48.972088 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24fcf\" (UniqueName: \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:49.060821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:49.060781 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" event={"ID":"54be536c-6fe9-45e4-9057-38bdc0e9d8a1","Type":"ContainerStarted","Data":"2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d"} Oct 30 06:22:49.088884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:49.088856 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:49.435381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:49.435162 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2"] Oct 30 06:22:49.450377 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:22:49.450337 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45dc8c9a_cb61_4983_8390_0416e2671f2f.slice/crio-1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad WatchSource:0}: Error finding container 1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad: Status 404 returned error can't find the container with id 1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad Oct 30 06:22:50.071367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:50.071330 6171 generic.go:356] "Generic (PLEG): container finished" podID="54be536c-6fe9-45e4-9057-38bdc0e9d8a1" containerID="f49ef91d11a3f11fc5dbe73655d7d65254fd4cf12e2259b11a38cc254ef3977b" exitCode=0 Oct 30 06:22:50.071553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:50.071414 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" event={"ID":"54be536c-6fe9-45e4-9057-38bdc0e9d8a1","Type":"ContainerDied","Data":"f49ef91d11a3f11fc5dbe73655d7d65254fd4cf12e2259b11a38cc254ef3977b"} Oct 30 06:22:50.073406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:50.073370 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" event={"ID":"45dc8c9a-cb61-4983-8390-0416e2671f2f","Type":"ContainerStarted","Data":"7338841c8133b4556d1c37daa6652e8814d1a604800d286aec1a0a3c8ed025a0"} Oct 30 06:22:50.073548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:50.073416 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" event={"ID":"45dc8c9a-cb61-4983-8390-0416e2671f2f","Type":"ContainerStarted","Data":"1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad"} Oct 30 06:22:50.160565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:50.160518 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" podStartSLOduration=2.160505547 podStartE2EDuration="2.160505547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:50.158469756 +0000 UTC m=+3028.598846948" watchObservedRunningTime="2026-10-30 06:22:50.160505547 +0000 UTC m=+3028.600882740" Oct 30 06:22:51.085163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.085024 6171 generic.go:356] "Generic (PLEG): container finished" podID="45dc8c9a-cb61-4983-8390-0416e2671f2f" containerID="7338841c8133b4556d1c37daa6652e8814d1a604800d286aec1a0a3c8ed025a0" exitCode=0 Oct 30 06:22:51.085639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.085167 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" event={"ID":"45dc8c9a-cb61-4983-8390-0416e2671f2f","Type":"ContainerDied","Data":"7338841c8133b4556d1c37daa6652e8814d1a604800d286aec1a0a3c8ed025a0"} Oct 30 06:22:51.094918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.086947 6171 generic.go:356] "Generic (PLEG): container finished" podID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerID="2d4f261b5342d60be847d5d6140f2ea8d81243aa1f5bc5492c6c4771e2591464" exitCode=0 Oct 30 06:22:51.094918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.087104 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" event={"ID":"89fc9606-370f-4fcb-b16c-5ec12af271b8","Type":"ContainerDied","Data":"2d4f261b5342d60be847d5d6140f2ea8d81243aa1f5bc5492c6c4771e2591464"} Oct 30 06:22:51.094918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.089251 6171 generic.go:356] "Generic (PLEG): container finished" podID="f5188865-8c7e-493e-806f-c98453ea8e94" containerID="d5fe23e5b4cb0cc28197a73097b170ee2fef39d81d4939936557a87201edb495" exitCode=0 Oct 30 06:22:51.094918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.089545 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" event={"ID":"f5188865-8c7e-493e-806f-c98453ea8e94","Type":"ContainerDied","Data":"d5fe23e5b4cb0cc28197a73097b170ee2fef39d81d4939936557a87201edb495"} Oct 30 06:22:51.215329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.215304 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:22:51.256224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.256199 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:22:51.266293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.266269 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:51.354061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354012 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\") pod \"89fc9606-370f-4fcb-b16c-5ec12af271b8\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " Oct 30 06:22:51.354258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354137 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\" (UniqueName: \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\") pod \"f5188865-8c7e-493e-806f-c98453ea8e94\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " Oct 30 06:22:51.354258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354217 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\") pod \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " Oct 30 06:22:51.354258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354252 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\") pod \"f5188865-8c7e-493e-806f-c98453ea8e94\" (UID: \"f5188865-8c7e-493e-806f-c98453ea8e94\") " Oct 30 06:22:51.354435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354286 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\" (UniqueName: \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\") pod \"89fc9606-370f-4fcb-b16c-5ec12af271b8\" (UID: \"89fc9606-370f-4fcb-b16c-5ec12af271b8\") " Oct 30 06:22:51.354435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354327 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\") pod \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " Oct 30 06:22:51.354546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354440 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume" pod "89fc9606-370f-4fcb-b16c-5ec12af271b8" (UID: "89fc9606-370f-4fcb-b16c-5ec12af271b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:51.354546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.354506 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\" (UniqueName: \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\") pod \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\" (UID: \"54be536c-6fe9-45e4-9057-38bdc0e9d8a1\") " Oct 30 06:22:51.355233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.355205 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89fc9606-370f-4fcb-b16c-5ec12af271b8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.356095 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.356071 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume" pod "f5188865-8c7e-493e-806f-c98453ea8e94" (UID: "f5188865-8c7e-493e-806f-c98453ea8e94"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:51.356253 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.356231 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca" pod "54be536c-6fe9-45e4-9057-38bdc0e9d8a1" (UID: "54be536c-6fe9-45e4-9057-38bdc0e9d8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:51.357406 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.357371 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca" pod "54be536c-6fe9-45e4-9057-38bdc0e9d8a1" (UID: "54be536c-6fe9-45e4-9057-38bdc0e9d8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:51.357698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.357675 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5" pod "89fc9606-370f-4fcb-b16c-5ec12af271b8" (UID: "89fc9606-370f-4fcb-b16c-5ec12af271b8"). InnerVolumeSpecName "kube-api-access-dklk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:51.357950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.357917 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j" pod "54be536c-6fe9-45e4-9057-38bdc0e9d8a1" (UID: "54be536c-6fe9-45e4-9057-38bdc0e9d8a1"). InnerVolumeSpecName "kube-api-access-r5h8j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:51.358651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.358630 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls" pod "f5188865-8c7e-493e-806f-c98453ea8e94" (UID: "f5188865-8c7e-493e-806f-c98453ea8e94"). InnerVolumeSpecName "kube-api-access-k9rls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.456949 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5h8j\" (UniqueName: \"kubernetes.io/projected/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-kube-api-access-r5h8j\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.456975 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9rls\" (UniqueName: \"kubernetes.io/projected/f5188865-8c7e-493e-806f-c98453ea8e94-kube-api-access-k9rls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.456985 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.456994 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f5188865-8c7e-493e-806f-c98453ea8e94-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.457004 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dklk5\" (UniqueName: \"kubernetes.io/projected/89fc9606-370f-4fcb-b16c-5ec12af271b8-kube-api-access-dklk5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:51.457019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:51.457013 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54be536c-6fe9-45e4-9057-38bdc0e9d8a1-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.102379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.102351 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" Oct 30 06:22:52.102832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.102376 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7" event={"ID":"54be536c-6fe9-45e4-9057-38bdc0e9d8a1","Type":"ContainerDied","Data":"2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d"} Oct 30 06:22:52.102832 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.102429 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2da10197b077a6c1de278e2a00b2521439813d474df40e6a7c79ad43331c625d" Oct 30 06:22:52.103943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.103922 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" Oct 30 06:22:52.104120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.103970 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz" event={"ID":"89fc9606-370f-4fcb-b16c-5ec12af271b8","Type":"ContainerDied","Data":"e62526a0a1cdcae9f4c0e4240a617cc30faffdc9abd22732f8ac24684c6762d5"} Oct 30 06:22:52.104120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.104020 6171 scope.go:122] "RemoveContainer" containerID="2d4f261b5342d60be847d5d6140f2ea8d81243aa1f5bc5492c6c4771e2591464" Oct 30 06:22:52.106351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.105908 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" Oct 30 06:22:52.106351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.105949 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9" event={"ID":"f5188865-8c7e-493e-806f-c98453ea8e94","Type":"ContainerDied","Data":"fd1d0a243c4ed39f6da92d49c488e3c6b8c6dde27142efc0b8c2dd0b06f89824"} Oct 30 06:22:52.116977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.116955 6171 scope.go:122] "RemoveContainer" containerID="d5fe23e5b4cb0cc28197a73097b170ee2fef39d81d4939936557a87201edb495" Oct 30 06:22:52.164643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.164578 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:22:52.184401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.184370 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-0f980e-698bd946d7-8gqcz"] Oct 30 06:22:52.216589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.216556 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:22:52.221342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.221318 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0f980e-b878b8d69-8mbv9"] Oct 30 06:22:52.267781 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.267757 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:52.368931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.368905 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\") pod \"45dc8c9a-cb61-4983-8390-0416e2671f2f\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " Oct 30 06:22:52.369127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.369010 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\" (UniqueName: \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\") pod \"45dc8c9a-cb61-4983-8390-0416e2671f2f\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " Oct 30 06:22:52.369222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.369208 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\") pod \"45dc8c9a-cb61-4983-8390-0416e2671f2f\" (UID: \"45dc8c9a-cb61-4983-8390-0416e2671f2f\") " Oct 30 06:22:52.369281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.369262 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca" pod "45dc8c9a-cb61-4983-8390-0416e2671f2f" (UID: "45dc8c9a-cb61-4983-8390-0416e2671f2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.369561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.369540 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.371353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.371330 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf" pod "45dc8c9a-cb61-4983-8390-0416e2671f2f" (UID: "45dc8c9a-cb61-4983-8390-0416e2671f2f"). InnerVolumeSpecName "kube-api-access-24fcf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.371451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.371328 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca" pod "45dc8c9a-cb61-4983-8390-0416e2671f2f" (UID: "45dc8c9a-cb61-4983-8390-0416e2671f2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:52.471332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.471307 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45dc8c9a-cb61-4983-8390-0416e2671f2f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.471332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.471332 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24fcf\" (UniqueName: \"kubernetes.io/projected/45dc8c9a-cb61-4983-8390-0416e2671f2f-kube-api-access-24fcf\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.662484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.662452 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:52.693643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.693617 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:22:52.775256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775225 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775275 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.775445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775305 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.775445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775359 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775415 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775455 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775482 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775527 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775593 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775629 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.775690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775654 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775787 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775827 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775878 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775923 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\") pod \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\" (UID: \"19f5fcf1-a619-494a-bf4f-ce031051a4f0\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.775976 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.776089 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776061 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.776414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776153 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\") pod \"cb9ebe3a-c3de-4555-8405-fb741f440345\" (UID: \"cb9ebe3a-c3de-4555-8405-fb741f440345\") " Oct 30 06:22:52.776414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776307 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.776414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776373 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.776593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776529 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.776656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776607 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776829 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.776992 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777012 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb9ebe3a-c3de-4555-8405-fb741f440345-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777026 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777080 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.777195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777096 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.777797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.777797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777385 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.777797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777466 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.777797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777675 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.777797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.777784 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.779659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.779623 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "kube-api-access-f98kv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.782254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.782231 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:52.782372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.782301 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "kube-api-access-fndn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.783979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.783936 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.783979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.783945 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:52.783979 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.783946 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token" pod "19f5fcf1-a619-494a-bf4f-ce031051a4f0" (UID: "19f5fcf1-a619-494a-bf4f-ce031051a4f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.784157 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.783977 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:52.785691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.785660 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy" pod "cb9ebe3a-c3de-4555-8405-fb741f440345" (UID: "cb9ebe3a-c3de-4555-8405-fb741f440345"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:52.854250 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.847544 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 06:22:52.870983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.870956 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 06:22:52.878591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878563 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878594 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878609 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878622 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878636 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878650 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f98kv\" (UniqueName: \"kubernetes.io/projected/cb9ebe3a-c3de-4555-8405-fb741f440345-kube-api-access-f98kv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878666 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878679 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878691 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878729 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19f5fcf1-a619-494a-bf4f-ce031051a4f0-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878743 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878757 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fndn4\" (UniqueName: \"kubernetes.io/projected/19f5fcf1-a619-494a-bf4f-ce031051a4f0-kube-api-access-fndn4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:52.878982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:52.878771 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb9ebe3a-c3de-4555-8405-fb741f440345-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:22:53.118822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.118730 6171 generic.go:356] "Generic (PLEG): container finished" podID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerID="e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f" exitCode=0 Oct 30 06:22:53.118822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.118795 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" Oct 30 06:22:53.119385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.118816 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerDied","Data":"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f"} Oct 30 06:22:53.119385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.118875 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc" event={"ID":"cb9ebe3a-c3de-4555-8405-fb741f440345","Type":"ContainerDied","Data":"0e7100fd7ae2b47cc0deb8055a681ac343c72ea49be3ecf8500dec81d9cbc749"} Oct 30 06:22:53.119385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.118904 6171 scope.go:122] "RemoveContainer" containerID="e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f" Oct 30 06:22:53.123402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.123377 6171 generic.go:356] "Generic (PLEG): container finished" podID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerID="5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588" exitCode=0 Oct 30 06:22:53.123537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.123454 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" Oct 30 06:22:53.123589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.123455 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" event={"ID":"19f5fcf1-a619-494a-bf4f-ce031051a4f0","Type":"ContainerDied","Data":"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588"} Oct 30 06:22:53.123589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.123577 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2" event={"ID":"19f5fcf1-a619-494a-bf4f-ce031051a4f0","Type":"ContainerDied","Data":"c91e1e6115a58fdee526cd77b3d19b77b0097a033ddaca0cb49ae3b6a813be57"} Oct 30 06:22:53.126550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.126528 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" Oct 30 06:22:53.126634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.126532 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2" event={"ID":"45dc8c9a-cb61-4983-8390-0416e2671f2f","Type":"ContainerDied","Data":"1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad"} Oct 30 06:22:53.126700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.126642 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fc822865ea23a69f881f3d8997908634fa745c80dfa1cb69224d2c6c7f094ad" Oct 30 06:22:53.134735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.134720 6171 scope.go:122] "RemoveContainer" containerID="beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176" Oct 30 06:22:53.154493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.154471 6171 scope.go:122] "RemoveContainer" containerID="e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f" Oct 30 06:22:53.154856 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:53.154820 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f\": container with ID starting with e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f not found: ID does not exist" containerID="e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f" Oct 30 06:22:53.154975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.154866 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f"} err="failed to get container status \"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f\": rpc error: code = NotFound desc = could not find container \"e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f\": container with ID starting with e5a6b6fc70ceac9cc390f0cac3080c6ae1dd7d9e047709b2b7f536837a03c17f not found: ID does not exist" Oct 30 06:22:53.154975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.154889 6171 scope.go:122] "RemoveContainer" containerID="beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176" Oct 30 06:22:53.155291 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:53.155269 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176\": container with ID starting with beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176 not found: ID does not exist" containerID="beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176" Oct 30 06:22:53.155333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.155304 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176"} err="failed to get container status \"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176\": rpc error: code = NotFound desc = could not find container \"beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176\": container with ID starting with beb2b388190b3eba456d1bbff34cad2902f74ef88066431da907719a98188176 not found: ID does not exist" Oct 30 06:22:53.155333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.155326 6171 scope.go:122] "RemoveContainer" containerID="5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588" Oct 30 06:22:53.166328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.166306 6171 scope.go:122] "RemoveContainer" containerID="5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588" Oct 30 06:22:53.166638 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:53.166618 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588\": container with ID starting with 5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588 not found: ID does not exist" containerID="5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588" Oct 30 06:22:53.166720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.166644 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588"} err="failed to get container status \"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588\": rpc error: code = NotFound desc = could not find container \"5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588\": container with ID starting with 5d1778d1329696445b285a24879da8e84e545608dedc00b06c0343c73f0c0588 not found: ID does not exist" Oct 30 06:22:53.180240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.180219 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:22:53.185928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.185907 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-k9xxc"] Oct 30 06:22:53.224458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.224428 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:53.228488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.228468 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-0f980e-openshift-default-5fb44cbb67-jb4s2"] Oct 30 06:22:53.672938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.672900 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" path="/var/lib/kubelet/pods/19f5fcf1-a619-494a-bf4f-ce031051a4f0/volumes" Oct 30 06:22:53.673396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.673381 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" path="/var/lib/kubelet/pods/89fc9606-370f-4fcb-b16c-5ec12af271b8/volumes" Oct 30 06:22:53.673821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.673807 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" path="/var/lib/kubelet/pods/cb9ebe3a-c3de-4555-8405-fb741f440345/volumes" Oct 30 06:22:53.674577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:53.674559 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" path="/var/lib/kubelet/pods/f5188865-8c7e-493e-806f-c98453ea8e94/volumes" Oct 30 06:22:56.101707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:56.101669 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:22:56.381004 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:22:56.380923 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:22:57.871967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:57.871920 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" containerID="cri-o://b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154" gracePeriod=25 Oct 30 06:22:58.610568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:58.610532 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 06:22:59.315553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:59.315516 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 06:22:59.352797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:22:59.352766 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 06:23:00.246054 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.246003 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:00.311673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.311629 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4"] Oct 30 06:23:00.313403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313377 6171 cpu_manager.go:414] "containerMap: removing container" podUID="54be536c-6fe9-45e4-9057-38bdc0e9d8a1" containerName="revoke-keys" Oct 30 06:23:00.313403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313401 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54be536c-6fe9-45e4-9057-38bdc0e9d8a1" containerName="revoke-keys" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313419 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313425 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313435 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313440 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313450 6171 cpu_manager.go:414] "containerMap: removing container" podUID="45dc8c9a-cb61-4983-8390-0416e2671f2f" containerName="revoke-keys" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313456 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45dc8c9a-cb61-4983-8390-0416e2671f2f" containerName="revoke-keys" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313469 6171 cpu_manager.go:414] "containerMap: removing container" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313474 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313486 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313493 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313529 6171 cpu_manager.go:414] "containerMap: removing container" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" Oct 30 06:23:00.313642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313534 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313658 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="45dc8c9a-cb61-4983-8390-0416e2671f2f" containerName="revoke-keys" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313670 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="19f5fcf1-a619-494a-bf4f-ce031051a4f0" containerName="istio-proxy" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313676 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313687 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fc9606-370f-4fcb-b16c-5ec12af271b8" containerName="payload-pre-processing" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313695 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="54be536c-6fe9-45e4-9057-38bdc0e9d8a1" containerName="revoke-keys" Oct 30 06:23:00.314365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.313701 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5188865-8c7e-493e-806f-c98453ea8e94" containerName="payload-processing" Oct 30 06:23:00.319573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.319538 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.326557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.326533 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd"] Oct 30 06:23:00.332521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.332496 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb9ebe3a-c3de-4555-8405-fb741f440345" containerName="istio-proxy" Oct 30 06:23:00.340137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.338213 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4"] Oct 30 06:23:00.340137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.338321 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.344073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.341774 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:00.344073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.341870 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:23:00.366428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.364083 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd"] Oct 30 06:23:00.413345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413308 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.413546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413352 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.413546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413380 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.413694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413540 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btzkl\" (UniqueName: \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.413694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413638 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.413694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.413654 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrp5t\" (UniqueName: \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.515076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.514985 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.515076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515021 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrp5t\" (UniqueName: \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.515311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515103 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.515311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515121 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.515311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515140 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.515311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515245 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btzkl\" (UniqueName: \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.515871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515816 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.515871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.515840 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.517753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.517731 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.517875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.517775 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.540678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.540656 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btzkl\" (UniqueName: \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\") pod \"maas-api-revoke-sub-36f3c7aa-hzxkd\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:00.540773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.540701 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrp5t\" (UniqueName: \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\") pod \"maas-api-revoke-sub-95eb23c9-q95d4\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.651486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.651449 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:00.661830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:00.661803 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:01.062265 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:01.062235 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60270d63_3a5e_4599_a2b7_f02abd45edef.slice/crio-79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423 WatchSource:0}: Error finding container 79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423: Status 404 returned error can't find the container with id 79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423 Oct 30 06:23:01.063286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:01.063261 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4"] Oct 30 06:23:01.075212 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:01.075178 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd"] Oct 30 06:23:01.076835 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:01.076812 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd3f2be_f258_4511_8b8c_f716b135563a.slice/crio-73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0 WatchSource:0}: Error finding container 73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0: Status 404 returned error can't find the container with id 73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0 Oct 30 06:23:01.321639 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:01.321568 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" event={"ID":"0dd3f2be-f258-4511-8b8c-f716b135563a","Type":"ContainerStarted","Data":"73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0"} Oct 30 06:23:01.323240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:01.323214 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" event={"ID":"60270d63-3a5e-4599-a2b7-f02abd45edef","Type":"ContainerStarted","Data":"79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423"} Oct 30 06:23:02.209547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.209315 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" containerID="cri-o://ea2376fae66628648ff1afad28a2d77b9945972a2cede9b60f4f7614d4e9c19f" gracePeriod=45 Oct 30 06:23:02.336161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.336132 6171 generic.go:356] "Generic (PLEG): container finished" podID="0dd3f2be-f258-4511-8b8c-f716b135563a" containerID="9765c29a3451bc19f623830d329edc6a8f9d6df7747d588cf57b739bb89c3439" exitCode=0 Oct 30 06:23:02.336583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.336232 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" event={"ID":"0dd3f2be-f258-4511-8b8c-f716b135563a","Type":"ContainerDied","Data":"9765c29a3451bc19f623830d329edc6a8f9d6df7747d588cf57b739bb89c3439"} Oct 30 06:23:02.338028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.338007 6171 generic.go:356] "Generic (PLEG): container finished" podID="60270d63-3a5e-4599-a2b7-f02abd45edef" containerID="2912d52867d14e65229c994911d100165fa492a0d8f195f0c32ad522870d263b" exitCode=0 Oct 30 06:23:02.338167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.338082 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" event={"ID":"60270d63-3a5e-4599-a2b7-f02abd45edef","Type":"ContainerDied","Data":"2912d52867d14e65229c994911d100165fa492a0d8f195f0c32ad522870d263b"} Oct 30 06:23:02.340023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.340003 6171 generic.go:356] "Generic (PLEG): container finished" podID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerID="ea2376fae66628648ff1afad28a2d77b9945972a2cede9b60f4f7614d4e9c19f" exitCode=0 Oct 30 06:23:02.340217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.340074 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" event={"ID":"9f4473e3-3e5c-4e5b-905b-cfe44b517f35","Type":"ContainerDied","Data":"ea2376fae66628648ff1afad28a2d77b9945972a2cede9b60f4f7614d4e9c19f"} Oct 30 06:23:02.541205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.541179 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:23:02.650699 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.650661 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.650910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.650750 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\" (UniqueName: \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.650910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.650891 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.651039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.650925 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.651039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.650982 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:02.651172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651087 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.651172 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651128 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\") pod \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\" (UID: \"9f4473e3-3e5c-4e5b-905b-cfe44b517f35\") " Oct 30 06:23:02.651416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651380 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:02.651617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651599 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:02.651806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651779 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.651882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651810 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.651882 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.651824 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.654533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.654504 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "kube-api-access-rvct5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:02.657216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.657186 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:02.658818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.658793 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm" pod "9f4473e3-3e5c-4e5b-905b-cfe44b517f35" (UID: "9f4473e3-3e5c-4e5b-905b-cfe44b517f35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:02.753387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.753354 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.753387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.753385 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvct5\" (UniqueName: \"kubernetes.io/projected/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-kube-api-access-rvct5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.753387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:02.753393 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f4473e3-3e5c-4e5b-905b-cfe44b517f35-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.349834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.349799 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" Oct 30 06:23:03.350392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.349832 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv" event={"ID":"9f4473e3-3e5c-4e5b-905b-cfe44b517f35","Type":"ContainerDied","Data":"803db64475fca1edcb0cb2d55a3baada1b7b64ed669c0ea6b236a9aa7023df71"} Oct 30 06:23:03.350392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.349887 6171 scope.go:122] "RemoveContainer" containerID="ea2376fae66628648ff1afad28a2d77b9945972a2cede9b60f4f7614d4e9c19f" Oct 30 06:23:03.413013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.412950 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 06:23:03.417247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.417181 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-1354a5/distinct-2-w6-1354a5-kserve-6b44d97df5-fs5pv"] Oct 30 06:23:03.580104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.580079 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:03.583479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.583460 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:03.611372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.611305 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" containerID="cri-o://446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9" gracePeriod=25 Oct 30 06:23:03.680688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.680649 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" path="/var/lib/kubelet/pods/9f4473e3-3e5c-4e5b-905b-cfe44b517f35/volumes" Oct 30 06:23:03.766339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766307 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\" (UniqueName: \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\") pod \"60270d63-3a5e-4599-a2b7-f02abd45edef\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " Oct 30 06:23:03.766339 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766344 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\") pod \"60270d63-3a5e-4599-a2b7-f02abd45edef\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " Oct 30 06:23:03.766600 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766550 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\") pod \"60270d63-3a5e-4599-a2b7-f02abd45edef\" (UID: \"60270d63-3a5e-4599-a2b7-f02abd45edef\") " Oct 30 06:23:03.766665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766646 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\") pod \"0dd3f2be-f258-4511-8b8c-f716b135563a\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " Oct 30 06:23:03.766712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766662 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca" pod "60270d63-3a5e-4599-a2b7-f02abd45edef" (UID: "60270d63-3a5e-4599-a2b7-f02abd45edef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:03.766776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766755 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\" (UniqueName: \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\") pod \"0dd3f2be-f258-4511-8b8c-f716b135563a\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " Oct 30 06:23:03.766923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.766902 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\") pod \"0dd3f2be-f258-4511-8b8c-f716b135563a\" (UID: \"0dd3f2be-f258-4511-8b8c-f716b135563a\") " Oct 30 06:23:03.767400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.767373 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca" pod "0dd3f2be-f258-4511-8b8c-f716b135563a" (UID: "0dd3f2be-f258-4511-8b8c-f716b135563a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:03.768016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.767989 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.768165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.768021 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.769075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.769027 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t" pod "60270d63-3a5e-4599-a2b7-f02abd45edef" (UID: "60270d63-3a5e-4599-a2b7-f02abd45edef"). InnerVolumeSpecName "kube-api-access-mrp5t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.769312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.769285 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca" pod "60270d63-3a5e-4599-a2b7-f02abd45edef" (UID: "60270d63-3a5e-4599-a2b7-f02abd45edef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:03.769574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.769553 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl" pod "0dd3f2be-f258-4511-8b8c-f716b135563a" (UID: "0dd3f2be-f258-4511-8b8c-f716b135563a"). InnerVolumeSpecName "kube-api-access-btzkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:03.770036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.770015 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca" pod "0dd3f2be-f258-4511-8b8c-f716b135563a" (UID: "0dd3f2be-f258-4511-8b8c-f716b135563a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:03.869355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.869277 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrp5t\" (UniqueName: \"kubernetes.io/projected/60270d63-3a5e-4599-a2b7-f02abd45edef-kube-api-access-mrp5t\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.869355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.869307 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60270d63-3a5e-4599-a2b7-f02abd45edef-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.869355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.869316 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0dd3f2be-f258-4511-8b8c-f716b135563a-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:03.869355 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:03.869324 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btzkl\" (UniqueName: \"kubernetes.io/projected/0dd3f2be-f258-4511-8b8c-f716b135563a-kube-api-access-btzkl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.316319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.316279 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" containerID="cri-o://5fbe7bf87e898e48eb7877d16dd2d601106d93bc1b8f31a7f23b8469c21c0c52" gracePeriod=25 Oct 30 06:23:04.353897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.353852 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" containerID="cri-o://c4c5255b38c5d10b2ed586267f5da2d1c83ffb33cf9e90d3f954773c37b49f74" gracePeriod=25 Oct 30 06:23:04.363213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.363183 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" Oct 30 06:23:04.363213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.363199 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd" event={"ID":"0dd3f2be-f258-4511-8b8c-f716b135563a","Type":"ContainerDied","Data":"73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0"} Oct 30 06:23:04.363382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.363257 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73e8f419de5dd2364cafcdff38bfda6d704e0208f1e821b6820d851d8065c0b0" Oct 30 06:23:04.364785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.364756 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" Oct 30 06:23:04.364920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.364802 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4" event={"ID":"60270d63-3a5e-4599-a2b7-f02abd45edef","Type":"ContainerDied","Data":"79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423"} Oct 30 06:23:04.364920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:04.364843 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79e18a710df2c0232b5eac357942126e85840e7cfb3ee6ef06f4050ce88dc423" Oct 30 06:23:07.395234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.395200 6171 generic.go:356] "Generic (PLEG): container finished" podID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerID="8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f" exitCode=0 Oct 30 06:23:07.395671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.395281 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerDied","Data":"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f"} Oct 30 06:23:07.395716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.395695 6171 scope.go:122] "RemoveContainer" containerID="8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f" Oct 30 06:23:07.397305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.397280 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:07.397379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.397343 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:07.397417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.397376 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:23:07.870878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.870843 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45"] Oct 30 06:23:07.871496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871481 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0dd3f2be-f258-4511-8b8c-f716b135563a" containerName="revoke-keys" Oct 30 06:23:07.871551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871497 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dd3f2be-f258-4511-8b8c-f716b135563a" containerName="revoke-keys" Oct 30 06:23:07.871551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871508 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" Oct 30 06:23:07.871551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871514 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" Oct 30 06:23:07.871651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871557 6171 cpu_manager.go:414] "containerMap: removing container" podUID="60270d63-3a5e-4599-a2b7-f02abd45edef" containerName="revoke-keys" Oct 30 06:23:07.871651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871563 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60270d63-3a5e-4599-a2b7-f02abd45edef" containerName="revoke-keys" Oct 30 06:23:07.871730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871720 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dd3f2be-f258-4511-8b8c-f716b135563a" containerName="revoke-keys" Oct 30 06:23:07.871770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871733 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f4473e3-3e5c-4e5b-905b-cfe44b517f35" containerName="main" Oct 30 06:23:07.871770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.871746 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="60270d63-3a5e-4599-a2b7-f02abd45edef" containerName="revoke-keys" Oct 30 06:23:07.875598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.875570 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:07.884091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.884038 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" containerID="cri-o://eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e" gracePeriod=45 Oct 30 06:23:07.893224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.893198 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:07.893396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.893379 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:23:07.894348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.894024 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt"] Oct 30 06:23:07.899650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.899626 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:07.906087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.905811 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45"] Oct 30 06:23:07.917294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.916910 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt"] Oct 30 06:23:07.919022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.918987 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z4tt\" (UniqueName: \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:07.919226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.919092 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:07.919226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.919204 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rn459\" (UniqueName: \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:07.919384 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.919257 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:07.920179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.920153 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:07.920286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:07.920215 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.021632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021578 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rn459\" (UniqueName: \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.021844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021658 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.021844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021724 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.021844 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021772 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.022069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021851 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z4tt\" (UniqueName: \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.022069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.021922 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.022773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.022740 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.023025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.022997 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.030209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.029398 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.031145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.031094 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.049482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.049449 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z4tt\" (UniqueName: \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\") pod \"maas-api-revoke-sub-8c2c519e-57b45\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.049648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.049449 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rn459\" (UniqueName: \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\") pod \"maas-api-revoke-sub-2e17309f-w8gqt\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.225169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.209285 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:08.272778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.272751 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:08.314833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.314808 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334598 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334679 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334725 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334780 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\" (UniqueName: \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334899 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.334954 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\") pod \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\" (UID: \"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788\") " Oct 30 06:23:08.335817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.335620 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.336221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.335847 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.336221 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.336000 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.337554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.337526 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:08.338297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.338260 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:08.338640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.338611 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff" pod "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" (UID: "d2ce5ad1-cd0d-41e5-a16c-b4f261b90788"). InnerVolumeSpecName "kube-api-access-nz7ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:08.424877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.424847 6171 generic.go:356] "Generic (PLEG): container finished" podID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerID="eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e" exitCode=0 Oct 30 06:23:08.425624 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.424891 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" Oct 30 06:23:08.425784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.424909 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" event={"ID":"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788","Type":"ContainerDied","Data":"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e"} Oct 30 06:23:08.425879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.425820 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" event={"ID":"d2ce5ad1-cd0d-41e5-a16c-b4f261b90788","Type":"ContainerDied","Data":"fc4bb32bee99a4c96ca5df8084e0ffd564598c4689cf3ab5771f37fd352f27d9"} Oct 30 06:23:08.425879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.425851 6171 scope.go:122] "RemoveContainer" containerID="eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e" Oct 30 06:23:08.430486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.429708 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerStarted","Data":"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0"} Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436420 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436437 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436444 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436452 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz7ff\" (UniqueName: \"kubernetes.io/projected/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-kube-api-access-nz7ff\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436462 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.437144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.436472 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:08.446229 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.445161 6171 scope.go:122] "RemoveContainer" containerID="eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e" Oct 30 06:23:08.446566 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:08.446497 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e\": container with ID starting with eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e not found: ID does not exist" containerID="eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e" Oct 30 06:23:08.446566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.446534 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e"} err="failed to get container status \"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e\": rpc error: code = NotFound desc = could not find container \"eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e\": container with ID starting with eed7b7112b9d45eb754cdc6f3e048b0f932eb82a46bb19ed83e59c4b5ab6f02e not found: ID does not exist" Oct 30 06:23:08.516136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.516102 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 06:23:08.522144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.522116 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn"] Oct 30 06:23:08.732622 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:08.732590 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d0e33cd_38e3_466d_89b0_9e667b3f4e9d.slice/crio-d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4 WatchSource:0}: Error finding container d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4: Status 404 returned error can't find the container with id d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4 Oct 30 06:23:08.745875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.745850 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45"] Oct 30 06:23:08.750437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.750412 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt"] Oct 30 06:23:08.751181 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:08.751152 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8c3a8e3_e819_4cf6_b439_c7aeec9f3969.slice/crio-3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1 WatchSource:0}: Error finding container 3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1: Status 404 returned error can't find the container with id 3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1 Oct 30 06:23:08.942771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.942696 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:23:08.943802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.943763 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" start-of-body= Oct 30 06:23:08.943962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:08.943813 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" Oct 30 06:23:09.011502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.011467 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2"] Oct 30 06:23:09.015692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.015667 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-c73fcb-9zzx2"] Oct 30 06:23:09.237125 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.237095 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:09.308825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.308784 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4a68a1/distinct-2-w1-4a68a1-kserve-8455fb98b4-bsgzn" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" probeResult="failure" output="Get \"https://10.133.0.88:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:23:09.441121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.441086 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" event={"ID":"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d","Type":"ContainerStarted","Data":"de3778d52d39f4f1e167d0dbf38563449adacff9ce85591872cf8007863b0ac0"} Oct 30 06:23:09.441571 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.441172 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" event={"ID":"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d","Type":"ContainerStarted","Data":"d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4"} Oct 30 06:23:09.442816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.442790 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" event={"ID":"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969","Type":"ContainerStarted","Data":"5194d1d778c32944ca34730cfcfcdf51e6aa985c2aa0b57f3814ec5265119aae"} Oct 30 06:23:09.442966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.442833 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" event={"ID":"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969","Type":"ContainerStarted","Data":"3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1"} Oct 30 06:23:09.509483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.509425 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" podStartSLOduration=2.509411945 podStartE2EDuration="2.509411945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:09.508382724 +0000 UTC m=+3047.948759921" watchObservedRunningTime="2026-10-30 06:23:09.509411945 +0000 UTC m=+3047.949789138" Oct 30 06:23:09.525493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.525461 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:09.525653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.525467 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:09.560873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.560831 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" podStartSLOduration=2.560818727 podStartE2EDuration="2.560818727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:09.556590368 +0000 UTC m=+3047.996967552" watchObservedRunningTime="2026-10-30 06:23:09.560818727 +0000 UTC m=+3048.001195920" Oct 30 06:23:09.673626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.673595 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45dc8c9a-cb61-4983-8390-0416e2671f2f" path="/var/lib/kubelet/pods/45dc8c9a-cb61-4983-8390-0416e2671f2f/volumes" Oct 30 06:23:09.674087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.674068 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" path="/var/lib/kubelet/pods/d2ce5ad1-cd0d-41e5-a16c-b4f261b90788/volumes" Oct 30 06:23:09.942911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.942879 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" start-of-body= Oct 30 06:23:09.943037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:09.942933 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" Oct 30 06:23:10.246463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.246422 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:10.453244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.453210 6171 generic.go:356] "Generic (PLEG): container finished" podID="b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" containerID="5194d1d778c32944ca34730cfcfcdf51e6aa985c2aa0b57f3814ec5265119aae" exitCode=0 Oct 30 06:23:10.453644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.453303 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" event={"ID":"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969","Type":"ContainerDied","Data":"5194d1d778c32944ca34730cfcfcdf51e6aa985c2aa0b57f3814ec5265119aae"} Oct 30 06:23:10.455165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.455140 6171 generic.go:356] "Generic (PLEG): container finished" podID="1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" containerID="de3778d52d39f4f1e167d0dbf38563449adacff9ce85591872cf8007863b0ac0" exitCode=0 Oct 30 06:23:10.455304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.455212 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" event={"ID":"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d","Type":"ContainerDied","Data":"de3778d52d39f4f1e167d0dbf38563449adacff9ce85591872cf8007863b0ac0"} Oct 30 06:23:10.811779 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.811736 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:23:10.942224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.942191 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" start-of-body= Oct 30 06:23:10.942425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:10.942249 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" Oct 30 06:23:11.824936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.824907 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:11.860247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.860224 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\") pod \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " Oct 30 06:23:11.860424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.860411 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\" (UniqueName: \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\") pod \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " Oct 30 06:23:11.860485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.860470 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\") pod \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\" (UID: \"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969\") " Oct 30 06:23:11.860604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.860579 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca" pod "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" (UID: "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:11.861018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.860997 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:11.862695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.862674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca" pod "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" (UID: "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:11.862792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.862743 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459" pod "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" (UID: "b8c3a8e3-e819-4cf6-b439-c7aeec9f3969"). InnerVolumeSpecName "kube-api-access-rn459". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:11.867015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.866989 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:11.942776 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.942740 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" start-of-body= Oct 30 06:23:11.942957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.942804 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" Oct 30 06:23:11.962270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.962243 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\") pod \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " Oct 30 06:23:11.962462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.962368 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\") pod \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " Oct 30 06:23:11.962559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.962504 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\" (UniqueName: \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\") pod \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\" (UID: \"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d\") " Oct 30 06:23:11.962701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.962664 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca" pod "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" (UID: "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:11.963405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.963379 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rn459\" (UniqueName: \"kubernetes.io/projected/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-kube-api-access-rn459\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:11.963515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.963409 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:11.963515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.963427 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:11.964737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.964708 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca" pod "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" (UID: "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:11.965577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:11.965560 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt" pod "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" (UID: "1d0e33cd-38e3-466d-89b0-9e667b3f4e9d"). InnerVolumeSpecName "kube-api-access-7z4tt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:12.064974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.064901 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.064974 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.064930 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z4tt\" (UniqueName: \"kubernetes.io/projected/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d-kube-api-access-7z4tt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.477029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.477001 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" Oct 30 06:23:12.477203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.477037 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45" event={"ID":"1d0e33cd-38e3-466d-89b0-9e667b3f4e9d","Type":"ContainerDied","Data":"d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4"} Oct 30 06:23:12.477203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.477131 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d282839f1712f216c6bf3fe97413a61f92b7e8048059cf2696bda590590e37c4" Oct 30 06:23:12.478661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.478641 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" Oct 30 06:23:12.478784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.478674 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt" event={"ID":"b8c3a8e3-e819-4cf6-b439-c7aeec9f3969","Type":"ContainerDied","Data":"3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1"} Oct 30 06:23:12.478784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.478716 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c98cf0741f5bd3225ab7168dc88ceba9a737c6be42718a6ef029c4c28212fa1" Oct 30 06:23:12.942480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.942448 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" start-of-body= Oct 30 06:23:12.942894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:12.942521 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.157:15021/healthz/ready\": dial tcp 10.133.1.157:15021: connect: connection refused" Oct 30 06:23:13.221879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:13.221853 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:23:14.088549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:14.088514 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:23:14.090315 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:14.090287 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:23:15.812209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:15.812155 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" podUID="ed11bf34-35ff-45f5-aee1-d262a9541428" containerName="payload-processing" containerID="cri-o://133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324" gracePeriod=25 Oct 30 06:23:16.087689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.087659 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:23:16.264123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.264092 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\") pod \"ed11bf34-35ff-45f5-aee1-d262a9541428\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " Oct 30 06:23:16.264353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.264325 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\" (UniqueName: \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\") pod \"ed11bf34-35ff-45f5-aee1-d262a9541428\" (UID: \"ed11bf34-35ff-45f5-aee1-d262a9541428\") " Oct 30 06:23:16.264494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.264473 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume" pod "ed11bf34-35ff-45f5-aee1-d262a9541428" (UID: "ed11bf34-35ff-45f5-aee1-d262a9541428"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:16.264887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.264870 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed11bf34-35ff-45f5-aee1-d262a9541428-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:16.266589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.266563 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9" pod "ed11bf34-35ff-45f5-aee1-d262a9541428" (UID: "ed11bf34-35ff-45f5-aee1-d262a9541428"). InnerVolumeSpecName "kube-api-access-stgc9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:16.366724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.366667 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stgc9\" (UniqueName: \"kubernetes.io/projected/ed11bf34-35ff-45f5-aee1-d262a9541428-kube-api-access-stgc9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:16.483037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.483004 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:23:16.483291 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.483267 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" containerID="cri-o://90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f" gracePeriod=30 Oct 30 06:23:16.538905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.538874 6171 generic.go:356] "Generic (PLEG): container finished" podID="ed11bf34-35ff-45f5-aee1-d262a9541428" containerID="133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324" exitCode=0 Oct 30 06:23:16.539058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.538917 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" Oct 30 06:23:16.539058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.538949 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" event={"ID":"ed11bf34-35ff-45f5-aee1-d262a9541428","Type":"ContainerDied","Data":"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324"} Oct 30 06:23:16.539058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.538997 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx" event={"ID":"ed11bf34-35ff-45f5-aee1-d262a9541428","Type":"ContainerDied","Data":"6aa768d678a7668a2d7b3b1320739a8daa39f98cb15d63d77c6482de6552fa6c"} Oct 30 06:23:16.539058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.539019 6171 scope.go:122] "RemoveContainer" containerID="133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324" Oct 30 06:23:16.551462 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.551446 6171 scope.go:122] "RemoveContainer" containerID="133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324" Oct 30 06:23:16.551781 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:16.551763 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324\": container with ID starting with 133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324 not found: ID does not exist" containerID="133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324" Oct 30 06:23:16.551841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.551791 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324"} err="failed to get container status \"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324\": rpc error: code = NotFound desc = could not find container \"133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324\": container with ID starting with 133cc3b36be95c2d6f1f9ce3398badc297530d9c28f04bd3609ad5487e27a324 not found: ID does not exist" Oct 30 06:23:16.670914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.670828 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:23:16.683893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:16.683740 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-5678c7df5c-hksbx"] Oct 30 06:23:17.369887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.369855 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8"] Oct 30 06:23:17.370528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370513 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" containerName="revoke-keys" Oct 30 06:23:17.370582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370530 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" containerName="revoke-keys" Oct 30 06:23:17.370582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370547 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" containerName="revoke-keys" Oct 30 06:23:17.370582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370552 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" containerName="revoke-keys" Oct 30 06:23:17.370582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370571 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" Oct 30 06:23:17.370582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370576 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370588 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ed11bf34-35ff-45f5-aee1-d262a9541428" containerName="payload-processing" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370594 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed11bf34-35ff-45f5-aee1-d262a9541428" containerName="payload-processing" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370701 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" containerName="revoke-keys" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370710 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" containerName="revoke-keys" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370720 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2ce5ad1-cd0d-41e5-a16c-b4f261b90788" containerName="main" Oct 30 06:23:17.370763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.370727 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed11bf34-35ff-45f5-aee1-d262a9541428" containerName="payload-processing" Oct 30 06:23:17.375597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.375577 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.379707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.379683 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.379838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.379712 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.379988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.379967 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w578m\" (UniqueName: \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.398097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.398072 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8"] Oct 30 06:23:17.480841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.480806 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w578m\" (UniqueName: \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.480989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.480880 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.480989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.480904 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.481667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.481646 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.483997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.483978 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:17.484105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.484033 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:23:17.494554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.494531 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.507921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.507898 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w578m\" (UniqueName: \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\") pod \"maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.672092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.672000 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed11bf34-35ff-45f5-aee1-d262a9541428" path="/var/lib/kubelet/pods/ed11bf34-35ff-45f5-aee1-d262a9541428/volumes" Oct 30 06:23:17.689264 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.689238 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:17.998254 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:17.998217 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a8222d9_de85_4488_989d_ddd9b95ba998.slice/crio-3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017 WatchSource:0}: Error finding container 3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017: Status 404 returned error can't find the container with id 3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017 Oct 30 06:23:17.998399 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:17.998379 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8"] Oct 30 06:23:18.134177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.134155 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 06:23:18.187839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.187805 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\") pod \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " Oct 30 06:23:18.188028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.187906 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\" (UniqueName: \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\") pod \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\" (UID: \"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2\") " Oct 30 06:23:18.188292 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.188268 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume" pod "4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" (UID: "4cbfcb39-a7e8-4a91-a196-639a1daa4dd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:18.188458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.188443 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.190237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.190213 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr" pod "4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" (UID: "4cbfcb39-a7e8-4a91-a196-639a1daa4dd2"). InnerVolumeSpecName "kube-api-access-7dcbr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:18.289373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.289312 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dcbr\" (UniqueName: \"kubernetes.io/projected/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2-kube-api-access-7dcbr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:18.557857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.557781 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" event={"ID":"0a8222d9-de85-4488-989d-ddd9b95ba998","Type":"ContainerStarted","Data":"273a8743fee85bc3a134c27b3699e4e85b95afb85f583f5204caec7a634c19fd"} Oct 30 06:23:18.557857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.557839 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" event={"ID":"0a8222d9-de85-4488-989d-ddd9b95ba998","Type":"ContainerStarted","Data":"3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017"} Oct 30 06:23:18.559380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.559353 6171 generic.go:356] "Generic (PLEG): container finished" podID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerID="b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154" exitCode=0 Oct 30 06:23:18.559502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.559382 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" event={"ID":"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2","Type":"ContainerDied","Data":"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154"} Oct 30 06:23:18.559502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.559430 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" event={"ID":"4cbfcb39-a7e8-4a91-a196-639a1daa4dd2","Type":"ContainerDied","Data":"5b0db99bea4194c257fff7e9396e42779eede3a334a4f8c7a177021fc18f4138"} Oct 30 06:23:18.559502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.559450 6171 scope.go:122] "RemoveContainer" containerID="b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154" Oct 30 06:23:18.559502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.559395 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t" Oct 30 06:23:18.571514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.571497 6171 scope.go:122] "RemoveContainer" containerID="b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154" Oct 30 06:23:18.571780 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:18.571762 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154\": container with ID starting with b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154 not found: ID does not exist" containerID="b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154" Oct 30 06:23:18.571853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.571786 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154"} err="failed to get container status \"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154\": rpc error: code = NotFound desc = could not find container \"b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154\": container with ID starting with b3d4826cbe82cf18f5de1d7d5bbbb20035b9dcf0eb10809f46549b71a4757154 not found: ID does not exist" Oct 30 06:23:18.603906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.603861 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" podStartSLOduration=1.603850253 podStartE2EDuration="1.603850253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:18.6017142 +0000 UTC m=+3057.042091393" watchObservedRunningTime="2026-10-30 06:23:18.603850253 +0000 UTC m=+3057.044227455" Oct 30 06:23:18.644362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.644329 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 06:23:18.648586 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:18.648567 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-1354a5-79877f4c99-ddp4t"] Oct 30 06:23:19.237873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.237836 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:19.526534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.526454 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:19.526534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.526477 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:19.570247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.570216 6171 generic.go:356] "Generic (PLEG): container finished" podID="0a8222d9-de85-4488-989d-ddd9b95ba998" containerID="273a8743fee85bc3a134c27b3699e4e85b95afb85f583f5204caec7a634c19fd" exitCode=0 Oct 30 06:23:19.570641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.570300 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" event={"ID":"0a8222d9-de85-4488-989d-ddd9b95ba998","Type":"ContainerDied","Data":"273a8743fee85bc3a134c27b3699e4e85b95afb85f583f5204caec7a634c19fd"} Oct 30 06:23:19.674633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:19.674602 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" path="/var/lib/kubelet/pods/4cbfcb39-a7e8-4a91-a196-639a1daa4dd2/volumes" Oct 30 06:23:20.747544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.747518 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:20.830061 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.830013 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\") pod \"0a8222d9-de85-4488-989d-ddd9b95ba998\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " Oct 30 06:23:20.830265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.830100 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\") pod \"0a8222d9-de85-4488-989d-ddd9b95ba998\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " Oct 30 06:23:20.830265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.830244 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\" (UniqueName: \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\") pod \"0a8222d9-de85-4488-989d-ddd9b95ba998\" (UID: \"0a8222d9-de85-4488-989d-ddd9b95ba998\") " Oct 30 06:23:20.830411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.830389 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca" pod "0a8222d9-de85-4488-989d-ddd9b95ba998" (UID: "0a8222d9-de85-4488-989d-ddd9b95ba998"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:20.830741 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.830717 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.832561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.832507 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca" pod "0a8222d9-de85-4488-989d-ddd9b95ba998" (UID: "0a8222d9-de85-4488-989d-ddd9b95ba998"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:20.832911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.832888 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m" pod "0a8222d9-de85-4488-989d-ddd9b95ba998" (UID: "0a8222d9-de85-4488-989d-ddd9b95ba998"). InnerVolumeSpecName "kube-api-access-w578m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.931761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.931736 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a8222d9-de85-4488-989d-ddd9b95ba998-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.931761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:20.931760 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w578m\" (UniqueName: \"kubernetes.io/projected/0a8222d9-de85-4488-989d-ddd9b95ba998-kube-api-access-w578m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.590217 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.590190 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" Oct 30 06:23:21.590372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.590218 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8" event={"ID":"0a8222d9-de85-4488-989d-ddd9b95ba998","Type":"ContainerDied","Data":"3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017"} Oct 30 06:23:21.590372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.590292 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a23444a8936cbc93c508cf2a43cd3f9dee0e988b27aa8e37ae3fecafe445017" Oct 30 06:23:21.755026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.754975 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:23:21.842298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842267 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842368 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842430 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842493 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842534 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842638 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842695 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842717 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:21.842805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842853 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\") pod \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\" (UID: \"bda2ddb7-892d-4784-8936-d5bfcafc98cf\") " Oct 30 06:23:21.842994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.842947 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:21.843177 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843157 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:21.843592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843561 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.843709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843595 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.843709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843611 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.843709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843690 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:21.844156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.843831 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:21.846137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.846113 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "kube-api-access-qcpvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:21.846416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.846392 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:21.846828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.846810 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:21.846950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.846923 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo" pod "bda2ddb7-892d-4784-8936-d5bfcafc98cf" (UID: "bda2ddb7-892d-4784-8936-d5bfcafc98cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:21.945023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.944997 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.945023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.945024 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.945216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.945033 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.945216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.945058 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcpvp\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-kube-api-access-qcpvp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.945216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.945073 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:21.945216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:21.945084 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bda2ddb7-892d-4784-8936-d5bfcafc98cf-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:22.600734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.600696 6171 generic.go:356] "Generic (PLEG): container finished" podID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerID="90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f" exitCode=0 Oct 30 06:23:22.600899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.600756 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" Oct 30 06:23:22.600899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.600783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" event={"ID":"bda2ddb7-892d-4784-8936-d5bfcafc98cf","Type":"ContainerDied","Data":"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f"} Oct 30 06:23:22.600899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.600837 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9" event={"ID":"bda2ddb7-892d-4784-8936-d5bfcafc98cf","Type":"ContainerDied","Data":"b1df7fe5d39ab85f005ac996691c60fcbfcb153d4c68ab8a69223505f08b8758"} Oct 30 06:23:22.600899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.600859 6171 scope.go:122] "RemoveContainer" containerID="90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f" Oct 30 06:23:22.615823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.615636 6171 scope.go:122] "RemoveContainer" containerID="90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f" Oct 30 06:23:22.615964 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:22.615938 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f\": container with ID starting with 90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f not found: ID does not exist" containerID="90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f" Oct 30 06:23:22.616018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.615976 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f"} err="failed to get container status \"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f\": rpc error: code = NotFound desc = could not find container \"90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f\": container with ID starting with 90a4f0d5694846bf4025756c96cc4c88f68cd670e8d584cffaa82f69efb8db9f not found: ID does not exist" Oct 30 06:23:22.674713 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.674688 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:23:22.688437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:22.688411 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-c73fcb-openshift-default-59c4764f55-sphv9"] Oct 30 06:23:23.672413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.672385 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" path="/var/lib/kubelet/pods/bda2ddb7-892d-4784-8936-d5bfcafc98cf/volumes" Oct 30 06:23:23.914560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.914537 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 06:23:23.986859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.986826 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\" (UniqueName: \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\") pod \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " Oct 30 06:23:23.987080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.986894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\") pod \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\" (UID: \"fd5a63a7-b570-42ba-ba6e-322ece01cea2\") " Oct 30 06:23:23.987338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.987310 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume" pod "fd5a63a7-b570-42ba-ba6e-322ece01cea2" (UID: "fd5a63a7-b570-42ba-ba6e-322ece01cea2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:23.989372 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:23.989343 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk" pod "fd5a63a7-b570-42ba-ba6e-322ece01cea2" (UID: "fd5a63a7-b570-42ba-ba6e-322ece01cea2"). InnerVolumeSpecName "kube-api-access-qqchk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:24.089131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.089102 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqchk\" (UniqueName: \"kubernetes.io/projected/fd5a63a7-b570-42ba-ba6e-322ece01cea2-kube-api-access-qqchk\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.089131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.089132 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd5a63a7-b570-42ba-ba6e-322ece01cea2-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.637572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.635037 6171 generic.go:356] "Generic (PLEG): container finished" podID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerID="446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9" exitCode=0 Oct 30 06:23:24.637572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.635199 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" Oct 30 06:23:24.637572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.635228 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" event={"ID":"fd5a63a7-b570-42ba-ba6e-322ece01cea2","Type":"ContainerDied","Data":"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9"} Oct 30 06:23:24.637572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.635281 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p" event={"ID":"fd5a63a7-b570-42ba-ba6e-322ece01cea2","Type":"ContainerDied","Data":"ad3b4ecf2334d415127e5237133ba917e8d1556d6e4fa234170e6af52a8e309e"} Oct 30 06:23:24.637572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.635307 6171 scope.go:122] "RemoveContainer" containerID="446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9" Oct 30 06:23:24.640171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.639198 6171 generic.go:356] "Generic (PLEG): container finished" podID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerID="c4c5255b38c5d10b2ed586267f5da2d1c83ffb33cf9e90d3f954773c37b49f74" exitCode=0 Oct 30 06:23:24.640171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.639385 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" event={"ID":"9e20720e-468f-4180-9e47-9dce2ec0f1c8","Type":"ContainerDied","Data":"c4c5255b38c5d10b2ed586267f5da2d1c83ffb33cf9e90d3f954773c37b49f74"} Oct 30 06:23:24.642066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.641218 6171 generic.go:356] "Generic (PLEG): container finished" podID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerID="5fbe7bf87e898e48eb7877d16dd2d601106d93bc1b8f31a7f23b8469c21c0c52" exitCode=0 Oct 30 06:23:24.642066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.641286 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" event={"ID":"209835cf-0fac-42a7-a57f-a13c76ddcff0","Type":"ContainerDied","Data":"5fbe7bf87e898e48eb7877d16dd2d601106d93bc1b8f31a7f23b8469c21c0c52"} Oct 30 06:23:24.652208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.652190 6171 scope.go:122] "RemoveContainer" containerID="446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9" Oct 30 06:23:24.652632 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:24.652600 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9\": container with ID starting with 446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9 not found: ID does not exist" containerID="446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9" Oct 30 06:23:24.652750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.652693 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9"} err="failed to get container status \"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9\": rpc error: code = NotFound desc = could not find container \"446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9\": container with ID starting with 446ae64b6127e30a114f585d47e48323e7dc1ab2f23b2493b060e54f362d90b9 not found: ID does not exist" Oct 30 06:23:24.718122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.718089 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 06:23:24.723001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.722970 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4a68a1-6fbfdb45f-wd46p"] Oct 30 06:23:24.729735 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.729709 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 06:23:24.815065 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.812507 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 06:23:24.826960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.820502 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\") pod \"209835cf-0fac-42a7-a57f-a13c76ddcff0\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " Oct 30 06:23:24.826960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.820706 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\" (UniqueName: \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\") pod \"209835cf-0fac-42a7-a57f-a13c76ddcff0\" (UID: \"209835cf-0fac-42a7-a57f-a13c76ddcff0\") " Oct 30 06:23:24.826960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.822110 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume" pod "209835cf-0fac-42a7-a57f-a13c76ddcff0" (UID: "209835cf-0fac-42a7-a57f-a13c76ddcff0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:24.826960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.825607 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct" pod "209835cf-0fac-42a7-a57f-a13c76ddcff0" (UID: "209835cf-0fac-42a7-a57f-a13c76ddcff0"). InnerVolumeSpecName "kube-api-access-gmpct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:24.922134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922019 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\") pod \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " Oct 30 06:23:24.922312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922239 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\" (UniqueName: \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\") pod \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\" (UID: \"9e20720e-468f-4180-9e47-9dce2ec0f1c8\") " Oct 30 06:23:24.922410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922385 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume" pod "9e20720e-468f-4180-9e47-9dce2ec0f1c8" (UID: "9e20720e-468f-4180-9e47-9dce2ec0f1c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:24.922694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922670 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/209835cf-0fac-42a7-a57f-a13c76ddcff0-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.922694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922692 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e20720e-468f-4180-9e47-9dce2ec0f1c8-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.922902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.922702 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmpct\" (UniqueName: \"kubernetes.io/projected/209835cf-0fac-42a7-a57f-a13c76ddcff0-kube-api-access-gmpct\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.924625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:24.924603 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd" pod "9e20720e-468f-4180-9e47-9dce2ec0f1c8" (UID: "9e20720e-468f-4180-9e47-9dce2ec0f1c8"). InnerVolumeSpecName "kube-api-access-hcmvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:25.028303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.028245 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcmvd\" (UniqueName: \"kubernetes.io/projected/9e20720e-468f-4180-9e47-9dce2ec0f1c8-kube-api-access-hcmvd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.652953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.652924 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" Oct 30 06:23:25.653176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.652974 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp" event={"ID":"9e20720e-468f-4180-9e47-9dce2ec0f1c8","Type":"ContainerDied","Data":"ee508ad786e73e68ea61f47efa275aa18aeca5ae514f5f192c443d9888564c65"} Oct 30 06:23:25.653176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.653113 6171 scope.go:122] "RemoveContainer" containerID="c4c5255b38c5d10b2ed586267f5da2d1c83ffb33cf9e90d3f954773c37b49f74" Oct 30 06:23:25.655130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.655026 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" event={"ID":"209835cf-0fac-42a7-a57f-a13c76ddcff0","Type":"ContainerDied","Data":"7b4f01f3536cc0949a73f033a7e25395306efd361410bc09891bf2ace9031f4d"} Oct 30 06:23:25.655130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.655035 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt" Oct 30 06:23:25.666798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.666552 6171 scope.go:122] "RemoveContainer" containerID="5fbe7bf87e898e48eb7877d16dd2d601106d93bc1b8f31a7f23b8469c21c0c52" Oct 30 06:23:25.672945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.672922 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" path="/var/lib/kubelet/pods/fd5a63a7-b570-42ba-ba6e-322ece01cea2/volumes" Oct 30 06:23:25.712747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.712723 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 06:23:25.726887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.726858 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-c73fcb-6488988655-tk7jp"] Oct 30 06:23:25.756435 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.756405 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 06:23:25.762636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.762615 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-c73fcb-6c99fdbb77-bmgzt"] Oct 30 06:23:25.974111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:25.974088 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:23:26.381638 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:26.381551 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:23:26.735709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.735676 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739016 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739058 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739090 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739098 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739117 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739125 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739154 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0a8222d9-de85-4488-989d-ddd9b95ba998" containerName="revoke-keys" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739161 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a8222d9-de85-4488-989d-ddd9b95ba998" containerName="revoke-keys" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739172 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739180 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739193 6171 cpu_manager.go:414] "containerMap: removing container" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" Oct 30 06:23:26.739227 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739201 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739345 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a8222d9-de85-4488-989d-ddd9b95ba998" containerName="revoke-keys" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739359 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda2ddb7-892d-4784-8936-d5bfcafc98cf" containerName="istio-proxy" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739369 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" containerName="payload-processing" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739387 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" containerName="payload-pre-processing" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739399 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd5a63a7-b570-42ba-ba6e-322ece01cea2" containerName="payload-pre-processing" Oct 30 06:23:26.739942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.739409 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbfcb39-a7e8-4a91-a196-639a1daa4dd2" containerName="payload-pre-processing" Oct 30 06:23:26.764743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.764714 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.773213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.773010 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:23:26.783331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.782824 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brckw\" (UniqueName: \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.783331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.782880 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.796729 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.796701 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1685dd-dockercfg-552sb\"" Oct 30 06:23:26.796958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.796936 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1685dd\"" Oct 30 06:23:26.884290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.884254 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brckw\" (UniqueName: \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.884486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.884315 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.884956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.884936 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.886973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.886951 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:23:26.891276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.891256 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:26.911805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.911774 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brckw\" (UniqueName: \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\") pod \"payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:26.913013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.912991 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:23:26.986064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.985975 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6zq8\" (UniqueName: \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:26.986064 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:26.986017 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.087312 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.087276 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6zq8\" (UniqueName: \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.087488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.087326 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.088104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.088086 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.089861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.089843 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:27.120962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.120901 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6zq8\" (UniqueName: \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\") pod \"payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.220066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.206275 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:27.536807 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:27.536776 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d43ead2_95cf_4ced_8f6b_1605d1b596c3.slice/crio-9c6eb4a946f968618ecf62adfb266c1f95e1a5ac97c14da7c788399c0644c25f WatchSource:0}: Error finding container 9c6eb4a946f968618ecf62adfb266c1f95e1a5ac97c14da7c788399c0644c25f: Status 404 returned error can't find the container with id 9c6eb4a946f968618ecf62adfb266c1f95e1a5ac97c14da7c788399c0644c25f Oct 30 06:23:27.537971 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.537946 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:23:27.669724 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:27.669692 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ff403f0_2d6f_4e7f_86a5_a6e30af2c91e.slice/crio-83abb5d828413195c28e18d82b068fd98b37abc5dff539786283c8e22cf12126 WatchSource:0}: Error finding container 83abb5d828413195c28e18d82b068fd98b37abc5dff539786283c8e22cf12126: Status 404 returned error can't find the container with id 83abb5d828413195c28e18d82b068fd98b37abc5dff539786283c8e22cf12126 Oct 30 06:23:27.675918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.675875 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="209835cf-0fac-42a7-a57f-a13c76ddcff0" path="/var/lib/kubelet/pods/209835cf-0fac-42a7-a57f-a13c76ddcff0/volumes" Oct 30 06:23:27.676308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.676292 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e20720e-468f-4180-9e47-9dce2ec0f1c8" path="/var/lib/kubelet/pods/9e20720e-468f-4180-9e47-9dce2ec0f1c8/volumes" Oct 30 06:23:27.676619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.676604 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:23:27.795222 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.794968 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" event={"ID":"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e","Type":"ContainerStarted","Data":"83abb5d828413195c28e18d82b068fd98b37abc5dff539786283c8e22cf12126"} Oct 30 06:23:27.812354 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.812295 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 06:23:27.816549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.816409 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" event={"ID":"6d43ead2-95cf-4ced-8f6b-1605d1b596c3","Type":"ContainerStarted","Data":"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0"} Oct 30 06:23:27.816549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.816503 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" event={"ID":"6d43ead2-95cf-4ced-8f6b-1605d1b596c3","Type":"ContainerStarted","Data":"9c6eb4a946f968618ecf62adfb266c1f95e1a5ac97c14da7c788399c0644c25f"} Oct 30 06:23:27.843923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.843885 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 06:23:27.974086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:27.974010 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" podStartSLOduration=1.973990955 podStartE2EDuration="1.973990955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:27.965103612 +0000 UTC m=+3066.405480806" watchObservedRunningTime="2026-10-30 06:23:27.973990955 +0000 UTC m=+3066.414368149" Oct 30 06:23:28.352210 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.352165 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s"] Oct 30 06:23:28.364615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.358169 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-1354a5-9x69s"] Oct 30 06:23:28.829433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.829384 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" event={"ID":"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e","Type":"ContainerStarted","Data":"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347"} Oct 30 06:23:28.829944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.829503 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:28.830004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.829952 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:28.878544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:28.878489 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" podStartSLOduration=2.878475149 podStartE2EDuration="2.878475149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:28.876410358 +0000 UTC m=+3067.316787551" watchObservedRunningTime="2026-10-30 06:23:28.878475149 +0000 UTC m=+3067.318852341" Oct 30 06:23:29.671825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:29.671792 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67e558d9-d6c2-490a-a3de-9384a45a7e34" path="/var/lib/kubelet/pods/67e558d9-d6c2-490a-a3de-9384a45a7e34/volumes" Oct 30 06:23:30.342405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:30.342371 6171 scope.go:122] "RemoveContainer" containerID="95eff2ef76e6b82df1ea1e104235402e61984c278625e892d84eb414ff9ec2e4" Oct 30 06:23:30.357915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:30.357891 6171 scope.go:122] "RemoveContainer" containerID="68f83adf23840e5aa952199dbd8adad15635e3635e58ed09458223a4c3e239a6" Oct 30 06:23:30.967296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:30.967216 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7"] Oct 30 06:23:30.981813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:30.981770 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-4a68a1-7lct7"] Oct 30 06:23:31.675099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:31.675004 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54be536c-6fe9-45e4-9057-38bdc0e9d8a1" path="/var/lib/kubelet/pods/54be536c-6fe9-45e4-9057-38bdc0e9d8a1/volumes" Oct 30 06:23:32.812989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:32.812920 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" containerID="cri-o://afc7bffa44bfb0d749b3f309ca38bcfecf7f08381f0f5fb8aa5a9489731f7189" gracePeriod=25 Oct 30 06:23:32.844671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:32.844617 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" containerID="cri-o://df7fe6473084770231a637b9241f2c2aa974ec00a7047d43197c2ee6e048aa64" gracePeriod=25 Oct 30 06:23:36.519559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:36.519523 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 06:23:36.520005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:36.519781 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" containerID="cri-o://db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1" gracePeriod=30 Oct 30 06:23:39.045394 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:39.045363 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 06:23:39.046018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:39.045613 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" containerID="cri-o://017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0" gracePeriod=30 Oct 30 06:23:39.840848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:39.840816 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:23:39.841018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:39.840888 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:23:41.254337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.254303 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:41.254337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.254322 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:41.800799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.800741 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 06:23:41.943785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.943745 6171 generic.go:356] "Generic (PLEG): container finished" podID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerID="db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1" exitCode=0 Oct 30 06:23:41.943985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.943796 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" Oct 30 06:23:41.943985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.943827 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" event={"ID":"f644c123-ad3b-47d6-a80c-3b160bd5edcc","Type":"ContainerDied","Data":"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1"} Oct 30 06:23:41.943985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.943880 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn" event={"ID":"f644c123-ad3b-47d6-a80c-3b160bd5edcc","Type":"ContainerDied","Data":"35aae412849d81582a59debecd5fa8bab31f80e7294012a368a56c689a4d6625"} Oct 30 06:23:41.943985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.943902 6171 scope.go:122] "RemoveContainer" containerID="db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1" Oct 30 06:23:41.955370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.955353 6171 scope.go:122] "RemoveContainer" containerID="db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1" Oct 30 06:23:41.955668 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:41.955650 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1\": container with ID starting with db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1 not found: ID does not exist" containerID="db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1" Oct 30 06:23:41.955727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.955678 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1"} err="failed to get container status \"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1\": rpc error: code = NotFound desc = could not find container \"db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1\": container with ID starting with db72307ccc4d4059e3c5e79e735f2e214e838bf93f82452168121519d43bedf1 not found: ID does not exist" Oct 30 06:23:41.962514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962489 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962547 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962596 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962655 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962678 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962749 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962797 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.962927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962818 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962883 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.962927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962920 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\") pod \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\" (UID: \"f644c123-ad3b-47d6-a80c-3b160bd5edcc\") " Oct 30 06:23:41.963188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962964 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.963188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.962981 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.963304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963249 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.963304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963177 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:41.963700 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963672 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.963774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963705 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.963774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963722 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.963774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963736 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.963774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.963749 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:41.964853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.964823 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:41.965015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.964995 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "kube-api-access-4c8rh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:41.965333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.965314 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:41.965698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:41.965683 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo" pod "f644c123-ad3b-47d6-a80c-3b160bd5edcc" (UID: "f644c123-ad3b-47d6-a80c-3b160bd5edcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:42.065123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.065039 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8rh\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-kube-api-access-4c8rh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:42.065123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.065094 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:42.065123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.065104 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:42.065123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.065113 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f644c123-ad3b-47d6-a80c-3b160bd5edcc-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:42.298027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.297993 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 06:23:42.312520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.312486 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-1354a5-openshift-default-6c46966c87-7c5bn"] Oct 30 06:23:42.656628 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.656599 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq"] Oct 30 06:23:42.657273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.657255 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" Oct 30 06:23:42.657273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.657272 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" Oct 30 06:23:42.657415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.657395 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" containerName="istio-proxy" Oct 30 06:23:42.662394 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.662376 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.680455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.680395 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5"] Oct 30 06:23:42.680695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.680673 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:23:42.702899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.700619 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:42.722348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.722315 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq"] Oct 30 06:23:42.722542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.722426 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.728623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.728588 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5"] Oct 30 06:23:42.771705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.771678 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.771881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.771860 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.771937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.771904 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95psh\" (UniqueName: \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.873513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873472 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.873679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873530 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6t7h\" (UniqueName: \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.873679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873560 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.873679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873585 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95psh\" (UniqueName: \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.873876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873833 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.873943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.873888 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.874176 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.874155 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.876335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.876310 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.898859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.898833 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95psh\" (UniqueName: \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\") pod \"maas-api-revoke-sub-cb1505f7-hmxdq\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.975934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.975898 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6t7h\" (UniqueName: \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.976127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.975967 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.976202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.976143 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.976739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.976710 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:42.977123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.977036 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:42.978985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:42.978965 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:43.009273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.009246 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6t7h\" (UniqueName: \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\") pod \"maas-api-revoke-sub-264cef08-7b7h5\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:43.020605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.020580 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:23:43.020810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.020789 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" containerID="cri-o://a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575" gracePeriod=30 Oct 30 06:23:43.048295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.048267 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:43.342629 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:43.342597 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podceaab93e_f29f_494c_9d8f_2a70112d4a06.slice/crio-f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0 WatchSource:0}: Error finding container f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0: Status 404 returned error can't find the container with id f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0 Oct 30 06:23:43.343021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.342776 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq"] Oct 30 06:23:43.459846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.459815 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5"] Oct 30 06:23:43.461654 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:23:43.461624 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487017ed_7702_4a96_abec_94fc1174a84d.slice/crio-46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2 WatchSource:0}: Error finding container 46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2: Status 404 returned error can't find the container with id 46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2 Oct 30 06:23:43.687627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.687592 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f644c123-ad3b-47d6-a80c-3b160bd5edcc" path="/var/lib/kubelet/pods/f644c123-ad3b-47d6-a80c-3b160bd5edcc/volumes" Oct 30 06:23:43.961511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.961480 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" event={"ID":"487017ed-7702-4a96-abec-94fc1174a84d","Type":"ContainerStarted","Data":"ae464467209683a151015be98902db215278fdc1b689a56381cfe248a89c4db7"} Oct 30 06:23:43.961666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.961536 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" event={"ID":"487017ed-7702-4a96-abec-94fc1174a84d","Type":"ContainerStarted","Data":"46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2"} Oct 30 06:23:43.963135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.963107 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" event={"ID":"ceaab93e-f29f-494c-9d8f-2a70112d4a06","Type":"ContainerStarted","Data":"a14d698d08124febf29b32f4c4f1a2d223b25d19b5b412301334603d5f8a2b53"} Oct 30 06:23:43.963231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:43.963158 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" event={"ID":"ceaab93e-f29f-494c-9d8f-2a70112d4a06","Type":"ContainerStarted","Data":"f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0"} Oct 30 06:23:44.007429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.007384 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" podStartSLOduration=2.007371017 podStartE2EDuration="2.007371017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:44.005354894 +0000 UTC m=+3082.445732087" watchObservedRunningTime="2026-10-30 06:23:44.007371017 +0000 UTC m=+3082.447748234" Oct 30 06:23:44.051911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.051859 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" podStartSLOduration=2.051846775 podStartE2EDuration="2.051846775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:44.051248956 +0000 UTC m=+3082.491626150" watchObservedRunningTime="2026-10-30 06:23:44.051846775 +0000 UTC m=+3082.492223999" Oct 30 06:23:44.313710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.313687 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 06:23:44.396097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396069 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396190 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396224 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396295 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396391 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396426 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.396528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396516 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.397021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396567 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\") pod \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\" (UID: \"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a\") " Oct 30 06:23:44.397130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396509 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.397130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396719 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.397130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.396734 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.397130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.397023 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:44.398164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.398113 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.398164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.398140 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.398164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.398156 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.398440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.398170 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.398440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.398184 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.399068 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.399015 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:44.399370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.399346 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "kube-api-access-98xkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:44.399433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.399391 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:44.399485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.399467 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo" pod "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" (UID: "eab53a95-fc61-4bae-b2e0-f3cb2cbc152a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:44.499656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.499633 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.499656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.499656 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.499801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.499665 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98xkw\" (UniqueName: \"kubernetes.io/projected/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-kube-api-access-98xkw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.499801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.499674 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:44.973970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.973936 6171 generic.go:356] "Generic (PLEG): container finished" podID="ceaab93e-f29f-494c-9d8f-2a70112d4a06" containerID="a14d698d08124febf29b32f4c4f1a2d223b25d19b5b412301334603d5f8a2b53" exitCode=0 Oct 30 06:23:44.974188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.974029 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" event={"ID":"ceaab93e-f29f-494c-9d8f-2a70112d4a06","Type":"ContainerDied","Data":"a14d698d08124febf29b32f4c4f1a2d223b25d19b5b412301334603d5f8a2b53"} Oct 30 06:23:44.975442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.975420 6171 generic.go:356] "Generic (PLEG): container finished" podID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerID="017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0" exitCode=0 Oct 30 06:23:44.975560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.975462 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" Oct 30 06:23:44.975560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.975501 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" event={"ID":"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a","Type":"ContainerDied","Data":"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0"} Oct 30 06:23:44.975560 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.975546 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt" event={"ID":"eab53a95-fc61-4bae-b2e0-f3cb2cbc152a","Type":"ContainerDied","Data":"98fbc1dc44935e4587126637faf51a7d6ba5c81c86bb9a196a7815b2eddecd3e"} Oct 30 06:23:44.975672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.975569 6171 scope.go:122] "RemoveContainer" containerID="017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0" Oct 30 06:23:44.977528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.977506 6171 generic.go:356] "Generic (PLEG): container finished" podID="487017ed-7702-4a96-abec-94fc1174a84d" containerID="ae464467209683a151015be98902db215278fdc1b689a56381cfe248a89c4db7" exitCode=0 Oct 30 06:23:44.977653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.977599 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" event={"ID":"487017ed-7702-4a96-abec-94fc1174a84d","Type":"ContainerDied","Data":"ae464467209683a151015be98902db215278fdc1b689a56381cfe248a89c4db7"} Oct 30 06:23:44.988489 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.988469 6171 scope.go:122] "RemoveContainer" containerID="017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0" Oct 30 06:23:44.988778 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:44.988759 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0\": container with ID starting with 017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0 not found: ID does not exist" containerID="017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0" Oct 30 06:23:44.988842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:44.988799 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0"} err="failed to get container status \"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0\": rpc error: code = NotFound desc = could not find container \"017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0\": container with ID starting with 017763b519ab40a0ecc07687b341b85885007b358649b05e9785b5c89c96e3e0 not found: ID does not exist" Oct 30 06:23:45.062213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:45.062183 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 06:23:45.075474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:45.075442 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4a68a1-openshift-default-64ff97f89c-55llt"] Oct 30 06:23:45.673604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:45.673569 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" path="/var/lib/kubelet/pods/eab53a95-fc61-4bae-b2e0-f3cb2cbc152a/volumes" Oct 30 06:23:46.149747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.149722 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:46.178458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.178435 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:46.235378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235347 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\") pod \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " Oct 30 06:23:46.235534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235435 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\" (UniqueName: \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\") pod \"487017ed-7702-4a96-abec-94fc1174a84d\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " Oct 30 06:23:46.235534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235461 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\") pod \"487017ed-7702-4a96-abec-94fc1174a84d\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " Oct 30 06:23:46.235534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235484 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\") pod \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " Oct 30 06:23:46.235852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235556 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\" (UniqueName: \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\") pod \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\" (UID: \"ceaab93e-f29f-494c-9d8f-2a70112d4a06\") " Oct 30 06:23:46.235852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235662 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\") pod \"487017ed-7702-4a96-abec-94fc1174a84d\" (UID: \"487017ed-7702-4a96-abec-94fc1174a84d\") " Oct 30 06:23:46.235852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.235786 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca" pod "ceaab93e-f29f-494c-9d8f-2a70112d4a06" (UID: "ceaab93e-f29f-494c-9d8f-2a70112d4a06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:46.236234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.236177 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca" pod "487017ed-7702-4a96-abec-94fc1174a84d" (UID: "487017ed-7702-4a96-abec-94fc1174a84d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:46.236440 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.236392 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.238019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.237954 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca" pod "ceaab93e-f29f-494c-9d8f-2a70112d4a06" (UID: "ceaab93e-f29f-494c-9d8f-2a70112d4a06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:46.238208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.238089 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca" pod "487017ed-7702-4a96-abec-94fc1174a84d" (UID: "487017ed-7702-4a96-abec-94fc1174a84d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:46.238404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.238351 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh" pod "ceaab93e-f29f-494c-9d8f-2a70112d4a06" (UID: "ceaab93e-f29f-494c-9d8f-2a70112d4a06"). InnerVolumeSpecName "kube-api-access-95psh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:46.238513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.238491 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h" pod "487017ed-7702-4a96-abec-94fc1174a84d" (UID: "487017ed-7702-4a96-abec-94fc1174a84d"). InnerVolumeSpecName "kube-api-access-t6t7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:46.337530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.337496 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95psh\" (UniqueName: \"kubernetes.io/projected/ceaab93e-f29f-494c-9d8f-2a70112d4a06-kube-api-access-95psh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.337530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.337525 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.337530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.337535 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6t7h\" (UniqueName: \"kubernetes.io/projected/487017ed-7702-4a96-abec-94fc1174a84d-kube-api-access-t6t7h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.337754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.337544 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/487017ed-7702-4a96-abec-94fc1174a84d-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.337754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.337553 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ceaab93e-f29f-494c-9d8f-2a70112d4a06-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:46.996278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.996248 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" Oct 30 06:23:46.996731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.996245 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5" event={"ID":"487017ed-7702-4a96-abec-94fc1174a84d","Type":"ContainerDied","Data":"46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2"} Oct 30 06:23:46.996731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.996422 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46960025c71b860b3f4415a16037ec26e0ddc73bd0afce2c32f1b0a3ee22aef2" Oct 30 06:23:46.997902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.997884 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" Oct 30 06:23:46.998032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.997929 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq" event={"ID":"ceaab93e-f29f-494c-9d8f-2a70112d4a06","Type":"ContainerDied","Data":"f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0"} Oct 30 06:23:46.998032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:46.997961 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f641a76af8188519180c960eb27208fe65cfe9a1113b1a38872d92419845d7b0" Oct 30 06:23:48.301849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.301815 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:23:48.359232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359202 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359244 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359351 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359412 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359484 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359484 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:48.359695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359603 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359630 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:48.359695 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359680 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359750 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359784 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\") pod \"1416e241-2058-4b57-bf24-3b616be2ce6a\" (UID: \"1416e241-2058-4b57-bf24-3b616be2ce6a\") " Oct 30 06:23:48.359987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.359961 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:48.360296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:48.360448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360318 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:48.360448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360355 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.360448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360369 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.360448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360378 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.360448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.360388 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.361675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.361654 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:48.361790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.361674 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:48.361883 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.361868 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "kube-api-access-whfsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:48.362280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.362265 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token" pod "1416e241-2058-4b57-bf24-3b616be2ce6a" (UID: "1416e241-2058-4b57-bf24-3b616be2ce6a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:48.461038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.461001 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.461038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.461031 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.461038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.461072 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1416e241-2058-4b57-bf24-3b616be2ce6a-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.461302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.461087 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whfsl\" (UniqueName: \"kubernetes.io/projected/1416e241-2058-4b57-bf24-3b616be2ce6a-kube-api-access-whfsl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:48.461302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:48.461100 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1416e241-2058-4b57-bf24-3b616be2ce6a-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:49.018643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.018607 6171 generic.go:356] "Generic (PLEG): container finished" podID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerID="a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575" exitCode=0 Oct 30 06:23:49.018836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.018667 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" Oct 30 06:23:49.018836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.018694 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerDied","Data":"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575"} Oct 30 06:23:49.018836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.018756 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm" event={"ID":"1416e241-2058-4b57-bf24-3b616be2ce6a","Type":"ContainerDied","Data":"3cd5578a835e43c72bf3e2496105782e7c6210ae4aa30b3c650d0dac3d647054"} Oct 30 06:23:49.018836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.018779 6171 scope.go:122] "RemoveContainer" containerID="a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575" Oct 30 06:23:49.031645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.031621 6171 scope.go:122] "RemoveContainer" containerID="2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686" Oct 30 06:23:49.042527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.042510 6171 scope.go:122] "RemoveContainer" containerID="a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575" Oct 30 06:23:49.042774 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:49.042753 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575\": container with ID starting with a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575 not found: ID does not exist" containerID="a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575" Oct 30 06:23:49.042848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.042780 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575"} err="failed to get container status \"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575\": rpc error: code = NotFound desc = could not find container \"a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575\": container with ID starting with a0db5e5530ed7a5b1dbe27bc05acb0daf1315708956d684a5c5239c7b87bd575 not found: ID does not exist" Oct 30 06:23:49.042848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.042799 6171 scope.go:122] "RemoveContainer" containerID="2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686" Oct 30 06:23:49.043091 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:49.043074 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686\": container with ID starting with 2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686 not found: ID does not exist" containerID="2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686" Oct 30 06:23:49.043141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.043097 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686"} err="failed to get container status \"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686\": rpc error: code = NotFound desc = could not find container \"2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686\": container with ID starting with 2315949f520fe5676b9800d89fe415e94fc5142da7d5e36484863dc562129686 not found: ID does not exist" Oct 30 06:23:49.080122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.080098 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:23:49.094869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.094842 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-bf87e1-openshift-default-85ff7b5ddf-7z9wm"] Oct 30 06:23:49.671683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:49.671651 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" path="/var/lib/kubelet/pods/1416e241-2058-4b57-bf24-3b616be2ce6a/volumes" Oct 30 06:23:51.253858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:51.253820 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:51.254299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:51.253832 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:53.062067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.061884 6171 generic.go:356] "Generic (PLEG): container finished" podID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerID="afc7bffa44bfb0d749b3f309ca38bcfecf7f08381f0f5fb8aa5a9489731f7189" exitCode=0 Oct 30 06:23:53.062067 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.061966 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" event={"ID":"9f16ffec-8432-4229-ab73-0c5ea222d7f9","Type":"ContainerDied","Data":"afc7bffa44bfb0d749b3f309ca38bcfecf7f08381f0f5fb8aa5a9489731f7189"} Oct 30 06:23:53.063880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.063858 6171 generic.go:356] "Generic (PLEG): container finished" podID="010116a0-d561-46eb-b06c-48c6988d1adc" containerID="df7fe6473084770231a637b9241f2c2aa974ec00a7047d43197c2ee6e048aa64" exitCode=0 Oct 30 06:23:53.064013 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.063892 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" event={"ID":"010116a0-d561-46eb-b06c-48c6988d1adc","Type":"ContainerDied","Data":"df7fe6473084770231a637b9241f2c2aa974ec00a7047d43197c2ee6e048aa64"} Oct 30 06:23:53.114842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.114817 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 06:23:53.144774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.144752 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 06:23:53.238359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.238315 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\") pod \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " Oct 30 06:23:53.238554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.238402 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\" (UniqueName: \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\") pod \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\" (UID: \"9f16ffec-8432-4229-ab73-0c5ea222d7f9\") " Oct 30 06:23:53.238554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.238533 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\" (UniqueName: \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\") pod \"010116a0-d561-46eb-b06c-48c6988d1adc\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " Oct 30 06:23:53.238683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.238609 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\") pod \"010116a0-d561-46eb-b06c-48c6988d1adc\" (UID: \"010116a0-d561-46eb-b06c-48c6988d1adc\") " Oct 30 06:23:53.238797 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.238766 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume" pod "9f16ffec-8432-4229-ab73-0c5ea222d7f9" (UID: "9f16ffec-8432-4229-ab73-0c5ea222d7f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:53.239032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.239008 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume" pod "010116a0-d561-46eb-b06c-48c6988d1adc" (UID: "010116a0-d561-46eb-b06c-48c6988d1adc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:53.239419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.239399 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/010116a0-d561-46eb-b06c-48c6988d1adc-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.239503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.239424 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f16ffec-8432-4229-ab73-0c5ea222d7f9-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.240842 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.240818 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw" pod "9f16ffec-8432-4229-ab73-0c5ea222d7f9" (UID: "9f16ffec-8432-4229-ab73-0c5ea222d7f9"). InnerVolumeSpecName "kube-api-access-6pzrw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:53.240929 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.240860 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx" pod "010116a0-d561-46eb-b06c-48c6988d1adc" (UID: "010116a0-d561-46eb-b06c-48c6988d1adc"). InnerVolumeSpecName "kube-api-access-w85tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:53.340966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.340904 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzrw\" (UniqueName: \"kubernetes.io/projected/9f16ffec-8432-4229-ab73-0c5ea222d7f9-kube-api-access-6pzrw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.340966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:53.340929 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w85tx\" (UniqueName: \"kubernetes.io/projected/010116a0-d561-46eb-b06c-48c6988d1adc-kube-api-access-w85tx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.073548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.073523 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" Oct 30 06:23:54.074099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.073564 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg" event={"ID":"9f16ffec-8432-4229-ab73-0c5ea222d7f9","Type":"ContainerDied","Data":"aafb06735403064c8096663bab89c082dba52c3440ea4206d27b7e9b363dd2d1"} Oct 30 06:23:54.074099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.073629 6171 scope.go:122] "RemoveContainer" containerID="afc7bffa44bfb0d749b3f309ca38bcfecf7f08381f0f5fb8aa5a9489731f7189" Oct 30 06:23:54.075313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.075296 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" Oct 30 06:23:54.075313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.075297 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk" event={"ID":"010116a0-d561-46eb-b06c-48c6988d1adc","Type":"ContainerDied","Data":"e7c51c3702376485bf56992aebbad932cd3431bd1f1e7d64391c8e611548a157"} Oct 30 06:23:54.107696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.107676 6171 scope.go:122] "RemoveContainer" containerID="df7fe6473084770231a637b9241f2c2aa974ec00a7047d43197c2ee6e048aa64" Oct 30 06:23:54.140969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.140944 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 06:23:54.154808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.154786 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-128436-79ddc6fd69-w4jsg"] Oct 30 06:23:54.183449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.183432 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 06:23:54.188413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:54.188392 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w3-128436-d84dd445f-hvbfk"] Oct 30 06:23:55.399568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.399533 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:23:55.400079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.399777 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" podUID="e93b6a22-40a9-4509-ae84-c77201323642" containerName="payload-pre-processing" containerID="cri-o://714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062" gracePeriod=30 Oct 30 06:23:55.675551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.675509 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" path="/var/lib/kubelet/pods/010116a0-d561-46eb-b06c-48c6988d1adc/volumes" Oct 30 06:23:55.676091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.676072 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" path="/var/lib/kubelet/pods/9f16ffec-8432-4229-ab73-0c5ea222d7f9/volumes" Oct 30 06:23:55.733930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.733889 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:23:55.873553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.873509 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\" (UniqueName: \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\") pod \"e93b6a22-40a9-4509-ae84-c77201323642\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " Oct 30 06:23:55.873553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.873553 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\") pod \"e93b6a22-40a9-4509-ae84-c77201323642\" (UID: \"e93b6a22-40a9-4509-ae84-c77201323642\") " Oct 30 06:23:55.873984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.873956 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume" pod "e93b6a22-40a9-4509-ae84-c77201323642" (UID: "e93b6a22-40a9-4509-ae84-c77201323642"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:55.875872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.875844 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms" pod "e93b6a22-40a9-4509-ae84-c77201323642" (UID: "e93b6a22-40a9-4509-ae84-c77201323642"). InnerVolumeSpecName "kube-api-access-j8cms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.975154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.975116 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e93b6a22-40a9-4509-ae84-c77201323642-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.975154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:55.975149 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8cms\" (UniqueName: \"kubernetes.io/projected/e93b6a22-40a9-4509-ae84-c77201323642-kube-api-access-j8cms\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:23:56.094870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.094835 6171 generic.go:356] "Generic (PLEG): container finished" podID="e93b6a22-40a9-4509-ae84-c77201323642" containerID="714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062" exitCode=0 Oct 30 06:23:56.095074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.094891 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" Oct 30 06:23:56.095074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.094919 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" event={"ID":"e93b6a22-40a9-4509-ae84-c77201323642","Type":"ContainerDied","Data":"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062"} Oct 30 06:23:56.095074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.094980 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26" event={"ID":"e93b6a22-40a9-4509-ae84-c77201323642","Type":"ContainerDied","Data":"01fee9d9f4ef24cfeb1ce09b8606a82f842e0b3042f4923978751dc54d24bf0d"} Oct 30 06:23:56.095074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.095009 6171 scope.go:122] "RemoveContainer" containerID="714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062" Oct 30 06:23:56.107678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.107652 6171 scope.go:122] "RemoveContainer" containerID="714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062" Oct 30 06:23:56.108173 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:56.108143 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062\": container with ID starting with 714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062 not found: ID does not exist" containerID="714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062" Oct 30 06:23:56.108289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.108180 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062"} err="failed to get container status \"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062\": rpc error: code = NotFound desc = could not find container \"714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062\": container with ID starting with 714b16871577a43d52186c4a91d4ba24328c55733980d297d2ad91de3b271062 not found: ID does not exist" Oct 30 06:23:56.154717 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.154555 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:23:56.159197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:56.159175 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d51cd8-5cc5bb6679-dgv26"] Oct 30 06:23:56.380605 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:23:56.380524 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:23:57.672984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:23:57.672952 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e93b6a22-40a9-4509-ae84-c77201323642" path="/var/lib/kubelet/pods/e93b6a22-40a9-4509-ae84-c77201323642/volumes" Oct 30 06:24:00.346325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:00.346290 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8"] Oct 30 06:24:00.352307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:00.352272 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-128436-vqhw8"] Oct 30 06:24:01.677911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:01.677869 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a8222d9-de85-4488-989d-ddd9b95ba998" path="/var/lib/kubelet/pods/0a8222d9-de85-4488-989d-ddd9b95ba998/volumes" Oct 30 06:24:03.422574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.422539 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:24:03.423036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.422778 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" containerID="cri-o://cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0" gracePeriod=30 Oct 30 06:24:03.436645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.436618 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:24:03.436873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.436848 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" containerID="cri-o://487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71" gracePeriod=30 Oct 30 06:24:03.440436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.440412 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:24:03.440645 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.440616 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" containerID="cri-o://4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf" gracePeriod=30 Oct 30 06:24:03.974156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974111 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs"] Oct 30 06:24:03.974712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974698 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974714 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974728 6171 cpu_manager.go:414] "containerMap: removing container" podUID="487017ed-7702-4a96-abec-94fc1174a84d" containerName="revoke-keys" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974733 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="487017ed-7702-4a96-abec-94fc1174a84d" containerName="revoke-keys" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974738 6171 cpu_manager.go:414] "containerMap: removing container" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974743 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" Oct 30 06:24:03.974761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974760 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974766 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974776 6171 cpu_manager.go:414] "containerMap: removing container" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974780 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974795 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ceaab93e-f29f-494c-9d8f-2a70112d4a06" containerName="revoke-keys" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974800 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ceaab93e-f29f-494c-9d8f-2a70112d4a06" containerName="revoke-keys" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974822 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e93b6a22-40a9-4509-ae84-c77201323642" containerName="payload-pre-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974827 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e93b6a22-40a9-4509-ae84-c77201323642" containerName="payload-pre-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974904 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974917 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e93b6a22-40a9-4509-ae84-c77201323642" containerName="payload-pre-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974928 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="487017ed-7702-4a96-abec-94fc1174a84d" containerName="revoke-keys" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974937 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="010116a0-d561-46eb-b06c-48c6988d1adc" containerName="payload-pre-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974945 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f16ffec-8432-4229-ab73-0c5ea222d7f9" containerName="payload-processing" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974951 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="eab53a95-fc61-4bae-b2e0-f3cb2cbc152a" containerName="istio-proxy" Oct 30 06:24:03.974962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.974958 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ceaab93e-f29f-494c-9d8f-2a70112d4a06" containerName="revoke-keys" Oct 30 06:24:03.979668 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.979648 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:03.997497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.997470 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:03.997803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:03.997783 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:24:04.000030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.000007 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs"] Oct 30 06:24:04.067850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.067810 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.068019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.067876 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.068019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.067986 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.168736 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.168705 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.168928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.168763 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.168928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.168802 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.169401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.169374 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.171341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.171311 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.195750 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.195720 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.292156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.292079 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:24:04.294693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.294667 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:04.605309 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:24:04.605276 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b84696d_36a3_42b7_84a0_efc889c00afc.slice/crio-0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428 WatchSource:0}: Error finding container 0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428: Status 404 returned error can't find the container with id 0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428 Oct 30 06:24:04.605712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.605354 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs"] Oct 30 06:24:04.918545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.918518 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:24:04.919141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.919117 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:04.919333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.919142 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:04.919333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.919287 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1416e241-2058-4b57-bf24-3b616be2ce6a" containerName="istio-proxy" Oct 30 06:24:04.922913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.922891 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:04.944563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.944544 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-630fdf-serving-cert\"" Oct 30 06:24:04.946899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:04.946878 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:24:05.081158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.081124 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.081347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.081182 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.081347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.081249 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.081434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.081356 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lb9v\" (UniqueName: \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.182508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.182417 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.182508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.182483 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.182742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.182555 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.182742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.182635 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lb9v\" (UniqueName: \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.183656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.183633 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.185734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.185710 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.185931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.185911 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.189437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.189408 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" event={"ID":"9b84696d-36a3-42b7-84a0-efc889c00afc","Type":"ContainerStarted","Data":"0ab805a9400a07b1bc26e734c8d37d4c2ece65315d55f9916a8fd575c0bd6de7"} Oct 30 06:24:05.189572 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.189488 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" event={"ID":"9b84696d-36a3-42b7-84a0-efc889c00afc","Type":"ContainerStarted","Data":"0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428"} Oct 30 06:24:05.208592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.208567 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lb9v\" (UniqueName: \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\") pod \"maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.237443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.237422 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:05.237618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.237580 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" podStartSLOduration=2.237568243 podStartE2EDuration="2.237568243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:05.23421658 +0000 UTC m=+3103.674593777" watchObservedRunningTime="2026-10-30 06:24:05.237568243 +0000 UTC m=+3103.677945436" Oct 30 06:24:05.547258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:05.547233 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:24:05.582059 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:24:05.582011 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08c677cf_faba_4522_bb38_4e6ded8f141b.slice/crio-5d6c8a9aaed945b82c7622319192dab17fc617232700041177bea625f310c959 WatchSource:0}: Error finding container 5d6c8a9aaed945b82c7622319192dab17fc617232700041177bea625f310c959: Status 404 returned error can't find the container with id 5d6c8a9aaed945b82c7622319192dab17fc617232700041177bea625f310c959 Oct 30 06:24:06.201565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.201537 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" event={"ID":"08c677cf-faba-4522-bb38-4e6ded8f141b","Type":"ContainerStarted","Data":"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2"} Oct 30 06:24:06.201936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.201592 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" event={"ID":"08c677cf-faba-4522-bb38-4e6ded8f141b","Type":"ContainerStarted","Data":"5d6c8a9aaed945b82c7622319192dab17fc617232700041177bea625f310c959"} Oct 30 06:24:06.201936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.201621 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:06.203333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.203308 6171 generic.go:356] "Generic (PLEG): container finished" podID="9b84696d-36a3-42b7-84a0-efc889c00afc" containerID="0ab805a9400a07b1bc26e734c8d37d4c2ece65315d55f9916a8fd575c0bd6de7" exitCode=0 Oct 30 06:24:06.203443 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.203355 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" event={"ID":"9b84696d-36a3-42b7-84a0-efc889c00afc","Type":"ContainerDied","Data":"0ab805a9400a07b1bc26e734c8d37d4c2ece65315d55f9916a8fd575c0bd6de7"} Oct 30 06:24:06.263163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:06.263118 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" podStartSLOduration=1.726899576 podStartE2EDuration="2.263104921s" totalImagesPullingTime="536.205345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:06.258981469 +0000 UTC m=+3104.699358667" watchObservedRunningTime="2026-10-30 06:24:06.263104921 +0000 UTC m=+3104.703482111" Oct 30 06:24:07.353148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.353119 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:07.514948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.514911 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\") pod \"9b84696d-36a3-42b7-84a0-efc889c00afc\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " Oct 30 06:24:07.515155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.515079 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\") pod \"9b84696d-36a3-42b7-84a0-efc889c00afc\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " Oct 30 06:24:07.515263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.515220 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\") pod \"9b84696d-36a3-42b7-84a0-efc889c00afc\" (UID: \"9b84696d-36a3-42b7-84a0-efc889c00afc\") " Oct 30 06:24:07.515608 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.515579 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca" pod "9b84696d-36a3-42b7-84a0-efc889c00afc" (UID: "9b84696d-36a3-42b7-84a0-efc889c00afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:07.517280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.517255 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca" pod "9b84696d-36a3-42b7-84a0-efc889c00afc" (UID: "9b84696d-36a3-42b7-84a0-efc889c00afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:07.517280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.517266 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb" pod "9b84696d-36a3-42b7-84a0-efc889c00afc" (UID: "9b84696d-36a3-42b7-84a0-efc889c00afc"). InnerVolumeSpecName "kube-api-access-qjppb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:07.616265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.616241 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:07.616265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.616265 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b84696d-36a3-42b7-84a0-efc889c00afc-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:07.616437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:07.616276 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjppb\" (UniqueName: \"kubernetes.io/projected/9b84696d-36a3-42b7-84a0-efc889c00afc-kube-api-access-qjppb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.221215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.221180 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" Oct 30 06:24:08.221421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.221187 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs" event={"ID":"9b84696d-36a3-42b7-84a0-efc889c00afc","Type":"ContainerDied","Data":"0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428"} Oct 30 06:24:08.221421 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.221356 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0230db02d2ebce17fb514c852c1dbe41fe484a0a87c565620abccceff12cf428" Oct 30 06:24:08.735185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.735156 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:24:08.746822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.746774 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:24:08.773455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.773435 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:24:08.829798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.829769 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.829948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.829836 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.829995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.829957 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830084 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830034 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830084 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830123 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830187 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.830282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830200 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830254 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830301 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\") pod \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\" (UID: \"fee094ed-58ff-4d39-99ff-8d7c9eb900b9\") " Oct 30 06:24:08.830446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830354 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.830563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830478 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.830772 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830730 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.830894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830796 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.830894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830858 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.830894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830877 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.830894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830892 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.831318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.830905 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.832687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.832660 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.832819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.832797 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.833299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.833269 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "kube-api-access-mcqf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.833400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.833319 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy" pod "fee094ed-58ff-4d39-99ff-8d7c9eb900b9" (UID: "fee094ed-58ff-4d39-99ff-8d7c9eb900b9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.931563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931525 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.931765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931594 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.931765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931624 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.931888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931827 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.931946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931921 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.932003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.931656 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932008 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932104 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932138 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932184 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932231 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932258 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932355 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932458 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932495 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932526 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932547 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.932581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932575 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\") pod \"0d136afb-70b6-4dd4-9baf-030e53aacb49\" (UID: \"0d136afb-70b6-4dd4-9baf-030e53aacb49\") " Oct 30 06:24:08.932811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932631 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.932988 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932937 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.933097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.933011 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.933591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.932663 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.933708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.933640 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\") pod \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\" (UID: \"09d31732-0dc7-4ae4-a935-9f48ead7bd24\") " Oct 30 06:24:08.933905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.933880 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.934357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934329 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934365 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934379 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934392 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934405 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d136afb-70b6-4dd4-9baf-030e53aacb49-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934418 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934431 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934444 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqf9\" (UniqueName: \"kubernetes.io/projected/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-kube-api-access-mcqf9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934460 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934475 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934489 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fee094ed-58ff-4d39-99ff-8d7c9eb900b9-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:08.934841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934818 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:08.935193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.934876 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.935193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.935088 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.935193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.935128 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.935376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.935311 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.935441 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.935415 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.935849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.935767 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:08.936382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.936361 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:08.936514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.936497 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "kube-api-access-2sh2m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.936902 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.936879 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token" pod "09d31732-0dc7-4ae4-a935-9f48ead7bd24" (UID: "09d31732-0dc7-4ae4-a935-9f48ead7bd24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.937433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.937413 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "kube-api-access-gmlwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:08.937698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:08.937681 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo" pod "0d136afb-70b6-4dd4-9baf-030e53aacb49" (UID: "0d136afb-70b6-4dd4-9baf-030e53aacb49"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035262 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035291 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035308 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035316 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035325 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmlwt\" (UniqueName: \"kubernetes.io/projected/0d136afb-70b6-4dd4-9baf-030e53aacb49-kube-api-access-gmlwt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035336 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d136afb-70b6-4dd4-9baf-030e53aacb49-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035344 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035353 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035361 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d136afb-70b6-4dd4-9baf-030e53aacb49-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035369 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sh2m\" (UniqueName: \"kubernetes.io/projected/09d31732-0dc7-4ae4-a935-9f48ead7bd24-kube-api-access-2sh2m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035378 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09d31732-0dc7-4ae4-a935-9f48ead7bd24-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.035650 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.035386 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09d31732-0dc7-4ae4-a935-9f48ead7bd24-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:09.231018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.230986 6171 generic.go:356] "Generic (PLEG): container finished" podID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerID="cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0" exitCode=0 Oct 30 06:24:09.231231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.231035 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" Oct 30 06:24:09.231231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.231081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" event={"ID":"09d31732-0dc7-4ae4-a935-9f48ead7bd24","Type":"ContainerDied","Data":"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0"} Oct 30 06:24:09.231231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.231169 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c" event={"ID":"09d31732-0dc7-4ae4-a935-9f48ead7bd24","Type":"ContainerDied","Data":"864dd5a81f946301e390741589fb21eb9af917c201c2c1693b683a311af35ccf"} Oct 30 06:24:09.231231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.231199 6171 scope.go:122] "RemoveContainer" containerID="cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0" Oct 30 06:24:09.232915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.232887 6171 generic.go:356] "Generic (PLEG): container finished" podID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerID="487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71" exitCode=0 Oct 30 06:24:09.233078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.232970 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" Oct 30 06:24:09.233078 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.232969 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" event={"ID":"fee094ed-58ff-4d39-99ff-8d7c9eb900b9","Type":"ContainerDied","Data":"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71"} Oct 30 06:24:09.233213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.233136 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54" event={"ID":"fee094ed-58ff-4d39-99ff-8d7c9eb900b9","Type":"ContainerDied","Data":"d1c19390c7b1f23aff93fb4da6e9db4d383ceb3e0b0ae2f4f7b41b2a8f4854f3"} Oct 30 06:24:09.234740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.234704 6171 generic.go:356] "Generic (PLEG): container finished" podID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerID="4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf" exitCode=0 Oct 30 06:24:09.234843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.234753 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" event={"ID":"0d136afb-70b6-4dd4-9baf-030e53aacb49","Type":"ContainerDied","Data":"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf"} Oct 30 06:24:09.234843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.234759 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" Oct 30 06:24:09.234843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.234787 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk" event={"ID":"0d136afb-70b6-4dd4-9baf-030e53aacb49","Type":"ContainerDied","Data":"3306be7e7823d042612b3dadef790391fbd4d4e2de7e6275def82acffd433ceb"} Oct 30 06:24:09.246765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.246741 6171 scope.go:122] "RemoveContainer" containerID="cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0" Oct 30 06:24:09.247103 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:09.247080 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0\": container with ID starting with cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0 not found: ID does not exist" containerID="cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0" Oct 30 06:24:09.247184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.247108 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0"} err="failed to get container status \"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0\": rpc error: code = NotFound desc = could not find container \"cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0\": container with ID starting with cf94ed61dffee77652d30384d62ee304729da097ceda7765ebf5b3451844fdb0 not found: ID does not exist" Oct 30 06:24:09.247184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.247124 6171 scope.go:122] "RemoveContainer" containerID="487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71" Oct 30 06:24:09.257521 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:09.257495 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d136afb_70b6_4dd4_9baf_030e53aacb49.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:24:09.258809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.258789 6171 scope.go:122] "RemoveContainer" containerID="487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71" Oct 30 06:24:09.259080 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:09.259039 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71\": container with ID starting with 487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71 not found: ID does not exist" containerID="487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71" Oct 30 06:24:09.259146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.259089 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71"} err="failed to get container status \"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71\": rpc error: code = NotFound desc = could not find container \"487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71\": container with ID starting with 487bdaff1b28ea2f81de3163513bc3cc08e28db2319590eddf8b6a8bd0017e71 not found: ID does not exist" Oct 30 06:24:09.259146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.259110 6171 scope.go:122] "RemoveContainer" containerID="4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf" Oct 30 06:24:09.269241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.269227 6171 scope.go:122] "RemoveContainer" containerID="4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf" Oct 30 06:24:09.269452 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:09.269435 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf\": container with ID starting with 4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf not found: ID does not exist" containerID="4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf" Oct 30 06:24:09.269505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.269459 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf"} err="failed to get container status \"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf\": rpc error: code = NotFound desc = could not find container \"4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf\": container with ID starting with 4acf92e134e6655fd7de207712c4321ddcd77c24abad130cd7fd79c9fcd7adcf not found: ID does not exist" Oct 30 06:24:09.308620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.308566 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:24:09.322150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.322127 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-m448c"] Oct 30 06:24:09.366011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.365984 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:24:09.370824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.370800 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-2rs54"] Oct 30 06:24:09.411605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.411581 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:24:09.415997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.415977 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-d51cd8-openshift-default-d6bcdb84c-frvrk"] Oct 30 06:24:09.672457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.672381 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" path="/var/lib/kubelet/pods/09d31732-0dc7-4ae4-a935-9f48ead7bd24/volumes" Oct 30 06:24:09.672877 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.672861 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" path="/var/lib/kubelet/pods/0d136afb-70b6-4dd4-9baf-030e53aacb49/volumes" Oct 30 06:24:09.673303 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:09.673289 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" path="/var/lib/kubelet/pods/fee094ed-58ff-4d39-99ff-8d7c9eb900b9/volumes" Oct 30 06:24:12.217810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:12.217777 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:24:14.384878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:14.384844 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 06:24:14.419031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:14.418998 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 06:24:16.095911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:16.095868 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:24:19.385866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:19.385827 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" containerID="cri-o://885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e" gracePeriod=25 Oct 30 06:24:19.420284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:19.420253 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" containerID="cri-o://b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0" gracePeriod=25 Oct 30 06:24:24.690430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:24.690392 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:24.690921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:24.690674 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:26.381160 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:26.381119 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:24:29.794732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:29.794697 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs"] Oct 30 06:24:29.808498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:29.808467 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-815776-s65vs"] Oct 30 06:24:30.034503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:30.034470 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq"] Oct 30 06:24:30.037617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:30.037594 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4a6368c-r62mq"] Oct 30 06:24:30.607538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:30.607505 6171 scope.go:122] "RemoveContainer" containerID="886bd73e625051ea0026cd3861d2aed88f04a32f71dc0ee2a4b72e7ecca9104e" Oct 30 06:24:31.671183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:31.671150 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61fadbac-a73e-44e1-b498-8937a3eb2957" path="/var/lib/kubelet/pods/61fadbac-a73e-44e1-b498-8937a3eb2957/volumes" Oct 30 06:24:31.671755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:31.671734 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b84696d-36a3-42b7-84a0-efc889c00afc" path="/var/lib/kubelet/pods/9b84696d-36a3-42b7-84a0-efc889c00afc/volumes" Oct 30 06:24:34.690033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:34.689986 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:34.690705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:34.690680 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:34.881982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:34.881932 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:24:37.340855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:37.340815 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:24:37.341376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:37.341102 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" containerID="cri-o://20fa3093f19c70e1acba6f485ba09d48fea0a1ede570b4f2da45ff0a96f458fa" gracePeriod=30 Oct 30 06:24:39.714859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.714834 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 06:24:39.748936 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.748913 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 06:24:39.868828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.868801 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\") pod \"890c1245-89a6-4e57-a78a-455e5aab5352\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " Oct 30 06:24:39.868998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.868911 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\") pod \"1dfd15d8-431d-44f1-835d-83e348a45fb9\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " Oct 30 06:24:39.868998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.868941 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\" (UniqueName: \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\") pod \"890c1245-89a6-4e57-a78a-455e5aab5352\" (UID: \"890c1245-89a6-4e57-a78a-455e5aab5352\") " Oct 30 06:24:39.869112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.869026 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\" (UniqueName: \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\") pod \"1dfd15d8-431d-44f1-835d-83e348a45fb9\" (UID: \"1dfd15d8-431d-44f1-835d-83e348a45fb9\") " Oct 30 06:24:39.869248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.869223 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume" pod "890c1245-89a6-4e57-a78a-455e5aab5352" (UID: "890c1245-89a6-4e57-a78a-455e5aab5352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:39.869364 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.869344 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume" pod "1dfd15d8-431d-44f1-835d-83e348a45fb9" (UID: "1dfd15d8-431d-44f1-835d-83e348a45fb9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:39.869439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.869426 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/890c1245-89a6-4e57-a78a-455e5aab5352-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:39.869480 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.869443 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1dfd15d8-431d-44f1-835d-83e348a45fb9-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:39.871265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.871246 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m" pod "890c1245-89a6-4e57-a78a-455e5aab5352" (UID: "890c1245-89a6-4e57-a78a-455e5aab5352"). InnerVolumeSpecName "kube-api-access-cfj6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:39.871368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.871345 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt" pod "1dfd15d8-431d-44f1-835d-83e348a45fb9" (UID: "1dfd15d8-431d-44f1-835d-83e348a45fb9"). InnerVolumeSpecName "kube-api-access-btgzt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:39.970702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.970640 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btgzt\" (UniqueName: \"kubernetes.io/projected/1dfd15d8-431d-44f1-835d-83e348a45fb9-kube-api-access-btgzt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:39.970702 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:39.970664 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfj6m\" (UniqueName: \"kubernetes.io/projected/890c1245-89a6-4e57-a78a-455e5aab5352-kube-api-access-cfj6m\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:40.496954 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.496922 6171 generic.go:356] "Generic (PLEG): container finished" podID="890c1245-89a6-4e57-a78a-455e5aab5352" containerID="885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e" exitCode=0 Oct 30 06:24:40.497135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.496985 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" Oct 30 06:24:40.497135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.497018 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" event={"ID":"890c1245-89a6-4e57-a78a-455e5aab5352","Type":"ContainerDied","Data":"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e"} Oct 30 06:24:40.497135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.497090 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt" event={"ID":"890c1245-89a6-4e57-a78a-455e5aab5352","Type":"ContainerDied","Data":"8c1293bb915fffb6d2d24ddd9a9cdfcba3196880b7fd2df8df4fe6875c6ab9fc"} Oct 30 06:24:40.497135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.497125 6171 scope.go:122] "RemoveContainer" containerID="885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e" Oct 30 06:24:40.498627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.498604 6171 generic.go:356] "Generic (PLEG): container finished" podID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerID="b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0" exitCode=0 Oct 30 06:24:40.498742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.498647 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" Oct 30 06:24:40.498742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.498713 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" event={"ID":"1dfd15d8-431d-44f1-835d-83e348a45fb9","Type":"ContainerDied","Data":"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0"} Oct 30 06:24:40.498859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.498774 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj" event={"ID":"1dfd15d8-431d-44f1-835d-83e348a45fb9","Type":"ContainerDied","Data":"4beb2a5b259bf008f39ad44e00b6abcf1df1f0b4c69543e422045160231e7022"} Oct 30 06:24:40.508114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.508094 6171 scope.go:122] "RemoveContainer" containerID="885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e" Oct 30 06:24:40.508419 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:40.508396 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e\": container with ID starting with 885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e not found: ID does not exist" containerID="885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e" Oct 30 06:24:40.508504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.508426 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e"} err="failed to get container status \"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e\": rpc error: code = NotFound desc = could not find container \"885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e\": container with ID starting with 885d960890055b7d46784deb079bdbf5569e97cc08a568dbde389b9b801c578e not found: ID does not exist" Oct 30 06:24:40.508504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.508442 6171 scope.go:122] "RemoveContainer" containerID="b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0" Oct 30 06:24:40.517929 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.517910 6171 scope.go:122] "RemoveContainer" containerID="b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0" Oct 30 06:24:40.518219 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:40.518197 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0\": container with ID starting with b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0 not found: ID does not exist" containerID="b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0" Oct 30 06:24:40.518286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.518225 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0"} err="failed to get container status \"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0\": rpc error: code = NotFound desc = could not find container \"b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0\": container with ID starting with b54daa967ce42db2d997c5f55f34a12ba5a638ebc39571ea4c37c6650e7f9da0 not found: ID does not exist" Oct 30 06:24:40.557730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.557697 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 06:24:40.574288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.573446 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-815776-7db74869c9-4z7bj"] Oct 30 06:24:40.601129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.601107 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 06:24:40.604323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:40.604302 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-815776-5d8f8d66b9-7lntt"] Oct 30 06:24:41.030466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:41.030433 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz"] Oct 30 06:24:41.033545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:41.033522 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9a9d6b1-q97pz"] Oct 30 06:24:41.672015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:41.671977 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" path="/var/lib/kubelet/pods/1dfd15d8-431d-44f1-835d-83e348a45fb9/volumes" Oct 30 06:24:41.672539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:41.672517 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372310df-6b38-4894-a936-3c64a4f086ef" path="/var/lib/kubelet/pods/372310df-6b38-4894-a936-3c64a4f086ef/volumes" Oct 30 06:24:41.672933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:41.672914 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" path="/var/lib/kubelet/pods/890c1245-89a6-4e57-a78a-455e5aab5352/volumes" Oct 30 06:24:42.523022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.522991 6171 generic.go:356] "Generic (PLEG): container finished" podID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerID="20fa3093f19c70e1acba6f485ba09d48fea0a1ede570b4f2da45ff0a96f458fa" exitCode=0 Oct 30 06:24:42.523410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.523081 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" event={"ID":"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef","Type":"ContainerDied","Data":"20fa3093f19c70e1acba6f485ba09d48fea0a1ede570b4f2da45ff0a96f458fa"} Oct 30 06:24:42.613705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.613646 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:24:42.698839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.698804 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.698981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.698852 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.698981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.698899 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.698981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.698937 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.698981 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.698964 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.699254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699072 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.699254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699125 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.699254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699226 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.699424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699296 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.699424 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699309 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\") pod \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\" (UID: \"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef\") " Oct 30 06:24:42.699583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.699758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699737 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.699833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699766 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.700000 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699979 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.700132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.700110 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.700186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.699914 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:42.702472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.702451 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.702564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.702468 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:42.702564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.702514 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.702795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.702767 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx" pod "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" (UID: "38fb093e-587f-4e18-bd9d-5b9b8b04f3ef"). InnerVolumeSpecName "kube-api-access-smwrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.801112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801079 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801108 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801120 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801132 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801143 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801175 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smwrx\" (UniqueName: \"kubernetes.io/projected/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-kube-api-access-smwrx\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.801282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:42.801188 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:43.531956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.531927 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" Oct 30 06:24:43.531956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.531943 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf" event={"ID":"38fb093e-587f-4e18-bd9d-5b9b8b04f3ef","Type":"ContainerDied","Data":"34878a20142f3175238bd5e5e57542e2b6211b92e3770b932ce2ab475a3f69b9"} Oct 30 06:24:43.532562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.532026 6171 scope.go:122] "RemoveContainer" containerID="20fa3093f19c70e1acba6f485ba09d48fea0a1ede570b4f2da45ff0a96f458fa" Oct 30 06:24:43.610886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.610863 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:24:43.624318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.624294 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-815776-openshift-default-64c6b9d5f9-c8lsf"] Oct 30 06:24:43.671960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:43.671933 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" path="/var/lib/kubelet/pods/38fb093e-587f-4e18-bd9d-5b9b8b04f3ef/volumes" Oct 30 06:24:48.303144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:48.303110 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 06:24:48.342887 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:48.342852 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 06:24:48.845240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:48.845203 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 06:24:48.845501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:48.845476 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" podUID="2e07cc31-169a-4507-8d3b-ff4476eb4519" containerName="maas-api" containerID="cri-o://7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf" gracePeriod=30 Oct 30 06:24:49.105083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.105012 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 06:24:49.166662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.166633 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\") pod \"2e07cc31-169a-4507-8d3b-ff4476eb4519\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " Oct 30 06:24:49.166808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.166696 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\") pod \"2e07cc31-169a-4507-8d3b-ff4476eb4519\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " Oct 30 06:24:49.166808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.166791 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\" (UniqueName: \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\") pod \"2e07cc31-169a-4507-8d3b-ff4476eb4519\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " Oct 30 06:24:49.167006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.166846 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\") pod \"2e07cc31-169a-4507-8d3b-ff4476eb4519\" (UID: \"2e07cc31-169a-4507-8d3b-ff4476eb4519\") " Oct 30 06:24:49.167091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.167017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca" pod "2e07cc31-169a-4507-8d3b-ff4476eb4519" (UID: "2e07cc31-169a-4507-8d3b-ff4476eb4519"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:49.167561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.167418 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e07cc31-169a-4507-8d3b-ff4476eb4519-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:49.169007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.168984 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls" pod "2e07cc31-169a-4507-8d3b-ff4476eb4519" (UID: "2e07cc31-169a-4507-8d3b-ff4476eb4519"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:49.169164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.169083 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn" pod "2e07cc31-169a-4507-8d3b-ff4476eb4519" (UID: "2e07cc31-169a-4507-8d3b-ff4476eb4519"). InnerVolumeSpecName "kube-api-access-dhbtn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:49.169294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.169274 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs" pod "2e07cc31-169a-4507-8d3b-ff4476eb4519" (UID: "2e07cc31-169a-4507-8d3b-ff4476eb4519"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:49.268246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.268217 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:49.268246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.268244 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhbtn\" (UniqueName: \"kubernetes.io/projected/2e07cc31-169a-4507-8d3b-ff4476eb4519-kube-api-access-dhbtn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:49.268404 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.268260 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e07cc31-169a-4507-8d3b-ff4476eb4519-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:24:49.580839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.580807 6171 generic.go:356] "Generic (PLEG): container finished" podID="2e07cc31-169a-4507-8d3b-ff4476eb4519" containerID="7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf" exitCode=0 Oct 30 06:24:49.581304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.580855 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" Oct 30 06:24:49.581304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.580901 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" event={"ID":"2e07cc31-169a-4507-8d3b-ff4476eb4519","Type":"ContainerDied","Data":"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf"} Oct 30 06:24:49.581304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.580961 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29" event={"ID":"2e07cc31-169a-4507-8d3b-ff4476eb4519","Type":"ContainerDied","Data":"f9e3827fc51a919b7a2cac3735ca17e59fe0903545979e84371d7e2739960b6a"} Oct 30 06:24:49.581304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.580985 6171 scope.go:122] "RemoveContainer" containerID="7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf" Oct 30 06:24:49.592499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.592479 6171 scope.go:122] "RemoveContainer" containerID="7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf" Oct 30 06:24:49.592757 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:49.592739 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf\": container with ID starting with 7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf not found: ID does not exist" containerID="7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf" Oct 30 06:24:49.592809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.592766 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf"} err="failed to get container status \"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf\": rpc error: code = NotFound desc = could not find container \"7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf\": container with ID starting with 7a19fdf62e140d2fdb4a3a1dd665ea0bc9e92d8a14245ecf759540dae3edc2bf not found: ID does not exist" Oct 30 06:24:49.639851 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.639819 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 06:24:49.653938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.653918 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-cc33d3-69f7cfc5df-svf29"] Oct 30 06:24:49.671609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:49.671581 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e07cc31-169a-4507-8d3b-ff4476eb4519" path="/var/lib/kubelet/pods/2e07cc31-169a-4507-8d3b-ff4476eb4519/volumes" Oct 30 06:24:53.303470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:53.303428 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" containerID="cri-o://d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56" gracePeriod=25 Oct 30 06:24:53.343827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:53.343791 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" containerID="cri-o://2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9" gracePeriod=25 Oct 30 06:24:54.042169 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:54.042120 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:24:55.372233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:55.372194 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:56.380057 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:24:56.380008 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:24:56.385273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:56.385237 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:59.992039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992007 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2"] Oct 30 06:24:59.992545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992531 6171 cpu_manager.go:414] "containerMap: removing container" podUID="9b84696d-36a3-42b7-84a0-efc889c00afc" containerName="revoke-keys" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992547 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b84696d-36a3-42b7-84a0-efc889c00afc" containerName="revoke-keys" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992560 6171 cpu_manager.go:414] "containerMap: removing container" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992566 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992572 6171 cpu_manager.go:414] "containerMap: removing container" podUID="2e07cc31-169a-4507-8d3b-ff4476eb4519" containerName="maas-api" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992577 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e07cc31-169a-4507-8d3b-ff4476eb4519" containerName="maas-api" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992599 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" Oct 30 06:24:59.992604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992604 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992613 6171 cpu_manager.go:414] "containerMap: removing container" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992618 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992632 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992638 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992651 6171 cpu_manager.go:414] "containerMap: removing container" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992657 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992668 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992674 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992760 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dfd15d8-431d-44f1-835d-83e348a45fb9" containerName="payload-pre-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992773 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b84696d-36a3-42b7-84a0-efc889c00afc" containerName="revoke-keys" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992779 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="09d31732-0dc7-4ae4-a935-9f48ead7bd24" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992786 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e07cc31-169a-4507-8d3b-ff4476eb4519" containerName="maas-api" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992792 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d136afb-70b6-4dd4-9baf-030e53aacb49" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992797 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="890c1245-89a6-4e57-a78a-455e5aab5352" containerName="payload-processing" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992804 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fee094ed-58ff-4d39-99ff-8d7c9eb900b9" containerName="istio-proxy" Oct 30 06:24:59.992858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.992809 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="38fb093e-587f-4e18-bd9d-5b9b8b04f3ef" containerName="istio-proxy" Oct 30 06:24:59.996319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:24:59.996303 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.015763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.015736 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2"] Oct 30 06:25:00.026197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.026174 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:00.026342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.026321 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:25:00.067627 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.067592 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wspsq\" (UniqueName: \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.067769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.067631 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.067820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.067759 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.168466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.168422 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wspsq\" (UniqueName: \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.168466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.168460 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.168738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.168517 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.169214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.169190 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.171024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.171001 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.193569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.193539 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wspsq\" (UniqueName: \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\") pod \"maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.310396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.310316 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:00.624783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.624751 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2"] Oct 30 06:25:00.624952 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:00.624764 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09dad874_e044_4a88_a1ef_790ee2bfe73a.slice/crio-b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375 WatchSource:0}: Error finding container b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375: Status 404 returned error can't find the container with id b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375 Oct 30 06:25:00.673371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:00.673335 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" event={"ID":"09dad874-e044-4a88-a1ef-790ee2bfe73a","Type":"ContainerStarted","Data":"b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375"} Oct 30 06:25:01.685233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:01.685145 6171 generic.go:356] "Generic (PLEG): container finished" podID="09dad874-e044-4a88-a1ef-790ee2bfe73a" containerID="2b3ff08a593016ef903a4a7146988447cfb21d8a19477fdbd6547b04a2b7f8ee" exitCode=0 Oct 30 06:25:01.685625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:01.685222 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" event={"ID":"09dad874-e044-4a88-a1ef-790ee2bfe73a","Type":"ContainerDied","Data":"2b3ff08a593016ef903a4a7146988447cfb21d8a19477fdbd6547b04a2b7f8ee"} Oct 30 06:25:02.830857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.830834 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:02.897661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.897627 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\") pod \"09dad874-e044-4a88-a1ef-790ee2bfe73a\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " Oct 30 06:25:02.897661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.897669 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\" (UniqueName: \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\") pod \"09dad874-e044-4a88-a1ef-790ee2bfe73a\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " Oct 30 06:25:02.897941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.897732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\") pod \"09dad874-e044-4a88-a1ef-790ee2bfe73a\" (UID: \"09dad874-e044-4a88-a1ef-790ee2bfe73a\") " Oct 30 06:25:02.898037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.897989 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca" pod "09dad874-e044-4a88-a1ef-790ee2bfe73a" (UID: "09dad874-e044-4a88-a1ef-790ee2bfe73a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:02.898199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.898180 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.900081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.900017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca" pod "09dad874-e044-4a88-a1ef-790ee2bfe73a" (UID: "09dad874-e044-4a88-a1ef-790ee2bfe73a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:02.900081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.900039 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq" pod "09dad874-e044-4a88-a1ef-790ee2bfe73a" (UID: "09dad874-e044-4a88-a1ef-790ee2bfe73a"). InnerVolumeSpecName "kube-api-access-wspsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:02.999304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.999268 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09dad874-e044-4a88-a1ef-790ee2bfe73a-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:02.999304 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:02.999296 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wspsq\" (UniqueName: \"kubernetes.io/projected/09dad874-e044-4a88-a1ef-790ee2bfe73a-kube-api-access-wspsq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:03.702294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:03.702263 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" Oct 30 06:25:03.702467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:03.702285 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2" event={"ID":"09dad874-e044-4a88-a1ef-790ee2bfe73a","Type":"ContainerDied","Data":"b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375"} Oct 30 06:25:03.702467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:03.702344 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5a8aec9e7aa282f7bfdf666e3c140f484a3425a6b9e4ca05bdb2a681658b375" Oct 30 06:25:05.372667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:05.372633 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:06.384689 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:06.384653 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:12.646175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.646141 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj"] Oct 30 06:25:12.646681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.646668 6171 cpu_manager.go:414] "containerMap: removing container" podUID="09dad874-e044-4a88-a1ef-790ee2bfe73a" containerName="revoke-keys" Oct 30 06:25:12.646681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.646682 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09dad874-e044-4a88-a1ef-790ee2bfe73a" containerName="revoke-keys" Oct 30 06:25:12.646773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.646757 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="09dad874-e044-4a88-a1ef-790ee2bfe73a" containerName="revoke-keys" Oct 30 06:25:12.650444 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.650424 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.684008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.683981 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:12.684187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.684164 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:25:12.687854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.687828 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj"] Oct 30 06:25:12.808733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.808701 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl2ck\" (UniqueName: \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.808881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.808781 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.808923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.808885 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.909756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.909681 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.909756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.909745 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl2ck\" (UniqueName: \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.909921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.909803 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.910418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.910397 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.912245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.912223 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.939010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.938991 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl2ck\" (UniqueName: \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\") pod \"maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:12.964108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:12.964087 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:13.275924 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.275892 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj"] Oct 30 06:25:13.276100 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:13.275888 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59f066db_398d_4399_b724_671357a1e3a4.slice/crio-98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe WatchSource:0}: Error finding container 98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe: Status 404 returned error can't find the container with id 98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe Oct 30 06:25:13.577705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.577674 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 06:25:13.624670 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.624647 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 06:25:13.718818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.718784 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\") pod \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " Oct 30 06:25:13.719338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.718861 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\") pod \"d7322401-c8ba-474a-817a-0558e3b62a1e\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " Oct 30 06:25:13.719338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.718915 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\" (UniqueName: \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\") pod \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\" (UID: \"ec4f5b3e-8631-4de5-a254-29e55f219a6b\") " Oct 30 06:25:13.719338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.718959 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\" (UniqueName: \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\") pod \"d7322401-c8ba-474a-817a-0558e3b62a1e\" (UID: \"d7322401-c8ba-474a-817a-0558e3b62a1e\") " Oct 30 06:25:13.719338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.719189 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume" pod "ec4f5b3e-8631-4de5-a254-29e55f219a6b" (UID: "ec4f5b3e-8631-4de5-a254-29e55f219a6b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:13.719338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.719267 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume" pod "d7322401-c8ba-474a-817a-0558e3b62a1e" (UID: "d7322401-c8ba-474a-817a-0558e3b62a1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:13.719654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.719531 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d7322401-c8ba-474a-817a-0558e3b62a1e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.719654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.719552 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec4f5b3e-8631-4de5-a254-29e55f219a6b-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.721242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.721219 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl" pod "ec4f5b3e-8631-4de5-a254-29e55f219a6b" (UID: "ec4f5b3e-8631-4de5-a254-29e55f219a6b"). InnerVolumeSpecName "kube-api-access-qjhsl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:13.721375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.721253 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq" pod "d7322401-c8ba-474a-817a-0558e3b62a1e" (UID: "d7322401-c8ba-474a-817a-0558e3b62a1e"). InnerVolumeSpecName "kube-api-access-vm8jq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:13.775651 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.775621 6171 generic.go:356] "Generic (PLEG): container finished" podID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerID="d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56" exitCode=0 Oct 30 06:25:13.775769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.775669 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" Oct 30 06:25:13.775769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.775705 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" event={"ID":"d7322401-c8ba-474a-817a-0558e3b62a1e","Type":"ContainerDied","Data":"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56"} Oct 30 06:25:13.775769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.775750 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5" event={"ID":"d7322401-c8ba-474a-817a-0558e3b62a1e","Type":"ContainerDied","Data":"301b3f57ddd2cd254399d6ae9d6b989a7636e8b1dd6faede07553421b7230fbc"} Oct 30 06:25:13.775904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.775771 6171 scope.go:122] "RemoveContainer" containerID="d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56" Oct 30 06:25:13.777615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.777593 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" event={"ID":"59f066db-398d-4399-b724-671357a1e3a4","Type":"ContainerStarted","Data":"d2356354aa5edf9e3e7fe7f7f11c1be184f7cd837c09f04dd74cda220d30a878"} Oct 30 06:25:13.777720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.777636 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" event={"ID":"59f066db-398d-4399-b724-671357a1e3a4","Type":"ContainerStarted","Data":"98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe"} Oct 30 06:25:13.779341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.779320 6171 generic.go:356] "Generic (PLEG): container finished" podID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerID="2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9" exitCode=0 Oct 30 06:25:13.779445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.779354 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" Oct 30 06:25:13.779445 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.779403 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" event={"ID":"ec4f5b3e-8631-4de5-a254-29e55f219a6b","Type":"ContainerDied","Data":"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9"} Oct 30 06:25:13.779538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.779444 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp" event={"ID":"ec4f5b3e-8631-4de5-a254-29e55f219a6b","Type":"ContainerDied","Data":"7a77e53137803202d8ebbef4f8f3663f1b46706752c5a57c75857e5ba0ad50b3"} Oct 30 06:25:13.789565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.789549 6171 scope.go:122] "RemoveContainer" containerID="d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56" Oct 30 06:25:13.789971 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:13.789939 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56\": container with ID starting with d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56 not found: ID does not exist" containerID="d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56" Oct 30 06:25:13.790117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.789980 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56"} err="failed to get container status \"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56\": rpc error: code = NotFound desc = could not find container \"d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56\": container with ID starting with d031a5463fe8be5e6e65549cbc0916f1e84136dcf108311a8cfb203b407dfa56 not found: ID does not exist" Oct 30 06:25:13.790117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.790003 6171 scope.go:122] "RemoveContainer" containerID="2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9" Oct 30 06:25:13.800186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.800169 6171 scope.go:122] "RemoveContainer" containerID="2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9" Oct 30 06:25:13.800450 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:13.800430 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9\": container with ID starting with 2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9 not found: ID does not exist" containerID="2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9" Oct 30 06:25:13.800537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.800453 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9"} err="failed to get container status \"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9\": rpc error: code = NotFound desc = could not find container \"2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9\": container with ID starting with 2bddd9ed795d8c87c823043e68ebbd881c6b695544c6f86b3b2ded6199ec29d9 not found: ID does not exist" Oct 30 06:25:13.820726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.820675 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjhsl\" (UniqueName: \"kubernetes.io/projected/ec4f5b3e-8631-4de5-a254-29e55f219a6b-kube-api-access-qjhsl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.820726 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.820699 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vm8jq\" (UniqueName: \"kubernetes.io/projected/d7322401-c8ba-474a-817a-0558e3b62a1e-kube-api-access-vm8jq\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:13.836833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.836810 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 06:25:13.839721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.839691 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-cc33d3-64b49c565f-rjltp"] Oct 30 06:25:13.880835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.880815 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 06:25:13.884782 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.884763 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-cc33d3-7bc47986fc-kd2q5"] Oct 30 06:25:13.934340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:13.934297 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" podStartSLOduration=1.934285911 podStartE2EDuration="1.934285911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:13.931971648 +0000 UTC m=+3172.372348839" watchObservedRunningTime="2026-10-30 06:25:13.934285911 +0000 UTC m=+3172.374663102" Oct 30 06:25:14.788209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:14.788172 6171 generic.go:356] "Generic (PLEG): container finished" podID="59f066db-398d-4399-b724-671357a1e3a4" containerID="d2356354aa5edf9e3e7fe7f7f11c1be184f7cd837c09f04dd74cda220d30a878" exitCode=0 Oct 30 06:25:14.789126 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:14.788272 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" event={"ID":"59f066db-398d-4399-b724-671357a1e3a4","Type":"ContainerDied","Data":"d2356354aa5edf9e3e7fe7f7f11c1be184f7cd837c09f04dd74cda220d30a878"} Oct 30 06:25:15.670484 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:15.670442 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" path="/var/lib/kubelet/pods/d7322401-c8ba-474a-817a-0558e3b62a1e/volumes" Oct 30 06:25:15.670916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:15.670895 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" path="/var/lib/kubelet/pods/ec4f5b3e-8631-4de5-a254-29e55f219a6b/volumes" Oct 30 06:25:15.944170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:15.944142 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:16.040184 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.040152 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\" (UniqueName: \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\") pod \"59f066db-398d-4399-b724-671357a1e3a4\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " Oct 30 06:25:16.040336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.040248 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\") pod \"59f066db-398d-4399-b724-671357a1e3a4\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " Oct 30 06:25:16.040336 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.040291 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\") pod \"59f066db-398d-4399-b724-671357a1e3a4\" (UID: \"59f066db-398d-4399-b724-671357a1e3a4\") " Oct 30 06:25:16.040595 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.040571 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca" pod "59f066db-398d-4399-b724-671357a1e3a4" (UID: "59f066db-398d-4399-b724-671357a1e3a4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:16.042512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.042489 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca" pod "59f066db-398d-4399-b724-671357a1e3a4" (UID: "59f066db-398d-4399-b724-671357a1e3a4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:16.042658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.042532 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck" pod "59f066db-398d-4399-b724-671357a1e3a4" (UID: "59f066db-398d-4399-b724-671357a1e3a4"). InnerVolumeSpecName "kube-api-access-fl2ck". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:16.141200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.141169 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl2ck\" (UniqueName: \"kubernetes.io/projected/59f066db-398d-4399-b724-671357a1e3a4-kube-api-access-fl2ck\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:16.141200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.141195 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59f066db-398d-4399-b724-671357a1e3a4-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:16.141397 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.141206 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59f066db-398d-4399-b724-671357a1e3a4-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:16.806492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.806455 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" event={"ID":"59f066db-398d-4399-b724-671357a1e3a4","Type":"ContainerDied","Data":"98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe"} Oct 30 06:25:16.806672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.806505 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bf790bb804ed7efd282a2570760a122c7011e08d1a4b7a082dda32c20134fe" Oct 30 06:25:16.806672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:16.806465 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj" Oct 30 06:25:17.411580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:17.411553 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 06:25:18.768427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.768399 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:25:18.768912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.768606 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" podUID="fed23fda-b84c-449c-8b28-3a73be116291" containerName="maas-api" containerID="cri-o://045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da" gracePeriod=30 Oct 30 06:25:18.928677 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.928634 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" podUID="fed23fda-b84c-449c-8b28-3a73be116291" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.164:8443/health\": dial tcp 10.133.1.164:8443: connect: connection refused" Oct 30 06:25:18.939038 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.939004 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:25:18.939267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.939228 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" podUID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" containerName="payload-processing" containerID="cri-o://514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347" gracePeriod=30 Oct 30 06:25:18.972144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.971307 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:25:18.972144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:18.971538 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" podUID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" containerName="payload-pre-processing" containerID="cri-o://6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0" gracePeriod=30 Oct 30 06:25:19.106755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.106719 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:25:19.247342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.247319 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:25:19.284603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.284506 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\") pod \"fed23fda-b84c-449c-8b28-3a73be116291\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " Oct 30 06:25:19.284785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.284632 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\") pod \"fed23fda-b84c-449c-8b28-3a73be116291\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " Oct 30 06:25:19.284785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.284739 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\" (UniqueName: \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\") pod \"fed23fda-b84c-449c-8b28-3a73be116291\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " Oct 30 06:25:19.284918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.284808 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\") pod \"fed23fda-b84c-449c-8b28-3a73be116291\" (UID: \"fed23fda-b84c-449c-8b28-3a73be116291\") " Oct 30 06:25:19.285826 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.285797 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca" pod "fed23fda-b84c-449c-8b28-3a73be116291" (UID: "fed23fda-b84c-449c-8b28-3a73be116291"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:19.287402 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.287376 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs" pod "fed23fda-b84c-449c-8b28-3a73be116291" (UID: "fed23fda-b84c-449c-8b28-3a73be116291"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:19.287539 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.287525 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls" pod "fed23fda-b84c-449c-8b28-3a73be116291" (UID: "fed23fda-b84c-449c-8b28-3a73be116291"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:19.288327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.288295 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6" pod "fed23fda-b84c-449c-8b28-3a73be116291" (UID: "fed23fda-b84c-449c-8b28-3a73be116291"). InnerVolumeSpecName "kube-api-access-k7sx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:19.304848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.304364 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:25:19.385656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.385626 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\" (UniqueName: \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\") pod \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " Oct 30 06:25:19.385876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.385714 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\") pod \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\" (UID: \"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e\") " Oct 30 06:25:19.386140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.386116 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.386220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.386142 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7sx6\" (UniqueName: \"kubernetes.io/projected/fed23fda-b84c-449c-8b28-3a73be116291-kube-api-access-k7sx6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.386220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.386136 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume" pod "3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" (UID: "3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:19.386220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.386155 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fed23fda-b84c-449c-8b28-3a73be116291-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.386220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.386167 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fed23fda-b84c-449c-8b28-3a73be116291-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.388276 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.388254 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8" pod "3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" (UID: "3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e"). InnerVolumeSpecName "kube-api-access-t6zq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:19.487614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.487583 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\") pod \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " Oct 30 06:25:19.487775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.487632 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\" (UniqueName: \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\") pod \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\" (UID: \"6d43ead2-95cf-4ced-8f6b-1605d1b596c3\") " Oct 30 06:25:19.487964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.487946 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6zq8\" (UniqueName: \"kubernetes.io/projected/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-kube-api-access-t6zq8\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.488030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.487971 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.488030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.488001 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume" pod "6d43ead2-95cf-4ced-8f6b-1605d1b596c3" (UID: "6d43ead2-95cf-4ced-8f6b-1605d1b596c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:19.489793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.489768 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw" pod "6d43ead2-95cf-4ced-8f6b-1605d1b596c3" (UID: "6d43ead2-95cf-4ced-8f6b-1605d1b596c3"). InnerVolumeSpecName "kube-api-access-brckw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:19.588740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.588678 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.588740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.588701 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brckw\" (UniqueName: \"kubernetes.io/projected/6d43ead2-95cf-4ced-8f6b-1605d1b596c3-kube-api-access-brckw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:19.622606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.622580 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2"] Oct 30 06:25:19.625447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.625427 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-1685dd-tfmp2"] Oct 30 06:25:19.682470 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.682430 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09dad874-e044-4a88-a1ef-790ee2bfe73a" path="/var/lib/kubelet/pods/09dad874-e044-4a88-a1ef-790ee2bfe73a/volumes" Oct 30 06:25:19.829506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.829479 6171 generic.go:356] "Generic (PLEG): container finished" podID="fed23fda-b84c-449c-8b28-3a73be116291" containerID="045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da" exitCode=0 Oct 30 06:25:19.829896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.829524 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" Oct 30 06:25:19.829896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.829546 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" event={"ID":"fed23fda-b84c-449c-8b28-3a73be116291","Type":"ContainerDied","Data":"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da"} Oct 30 06:25:19.829896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.829584 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j" event={"ID":"fed23fda-b84c-449c-8b28-3a73be116291","Type":"ContainerDied","Data":"82f821f84022ccb3e4a4f4579cc18bbdac9516560d21754c31147fa0548f49a9"} Oct 30 06:25:19.829896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.829613 6171 scope.go:122] "RemoveContainer" containerID="045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da" Oct 30 06:25:19.831161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.831129 6171 generic.go:356] "Generic (PLEG): container finished" podID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" containerID="514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347" exitCode=0 Oct 30 06:25:19.831289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.831177 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" Oct 30 06:25:19.831289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.831210 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" event={"ID":"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e","Type":"ContainerDied","Data":"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347"} Oct 30 06:25:19.831289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.831262 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq" event={"ID":"3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e","Type":"ContainerDied","Data":"83abb5d828413195c28e18d82b068fd98b37abc5dff539786283c8e22cf12126"} Oct 30 06:25:19.832675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.832656 6171 generic.go:356] "Generic (PLEG): container finished" podID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" containerID="6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0" exitCode=0 Oct 30 06:25:19.832769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.832700 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" Oct 30 06:25:19.832769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.832717 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" event={"ID":"6d43ead2-95cf-4ced-8f6b-1605d1b596c3","Type":"ContainerDied","Data":"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0"} Oct 30 06:25:19.832769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.832760 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm" event={"ID":"6d43ead2-95cf-4ced-8f6b-1605d1b596c3","Type":"ContainerDied","Data":"9c6eb4a946f968618ecf62adfb266c1f95e1a5ac97c14da7c788399c0644c25f"} Oct 30 06:25:19.841101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.841082 6171 scope.go:122] "RemoveContainer" containerID="045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da" Oct 30 06:25:19.841312 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:19.841295 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da\": container with ID starting with 045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da not found: ID does not exist" containerID="045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da" Oct 30 06:25:19.841369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.841328 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da"} err="failed to get container status \"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da\": rpc error: code = NotFound desc = could not find container \"045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da\": container with ID starting with 045524a12bb2ac68f916977e1145dd1f2c675afbe7d82cbc6d1dc0959f2526da not found: ID does not exist" Oct 30 06:25:19.841369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.841344 6171 scope.go:122] "RemoveContainer" containerID="514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347" Oct 30 06:25:19.850423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.850407 6171 scope.go:122] "RemoveContainer" containerID="514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347" Oct 30 06:25:19.850674 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:19.850658 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347\": container with ID starting with 514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347 not found: ID does not exist" containerID="514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347" Oct 30 06:25:19.850719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.850680 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347"} err="failed to get container status \"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347\": rpc error: code = NotFound desc = could not find container \"514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347\": container with ID starting with 514d1b800b60002b0c6c2a93d8fd08c559edd2a67591027ac50a3ce58b4d7347 not found: ID does not exist" Oct 30 06:25:19.850719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.850695 6171 scope.go:122] "RemoveContainer" containerID="6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0" Oct 30 06:25:19.859755 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.859739 6171 scope.go:122] "RemoveContainer" containerID="6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0" Oct 30 06:25:19.859996 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:19.859977 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0\": container with ID starting with 6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0 not found: ID does not exist" containerID="6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0" Oct 30 06:25:19.860100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.860005 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0"} err="failed to get container status \"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0\": rpc error: code = NotFound desc = could not find container \"6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0\": container with ID starting with 6df294f518ca4379b610ee9f8154cc788ed3e3df9e13c8cbedac72076a6e5bb0 not found: ID does not exist" Oct 30 06:25:19.890879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.890856 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:25:19.895927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.895907 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1685dd-698c565cb7-8s8pm"] Oct 30 06:25:19.942656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.942543 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:25:19.950337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.950308 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-1685dd-7dd48cb8db-mgznq"] Oct 30 06:25:19.985081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.985037 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:25:19.988592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:19.988570 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-1685dd-856c9558d9-vb44j"] Oct 30 06:25:21.671798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:21.671765 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" path="/var/lib/kubelet/pods/3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e/volumes" Oct 30 06:25:21.672334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:21.672312 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" path="/var/lib/kubelet/pods/6d43ead2-95cf-4ced-8f6b-1605d1b596c3/volumes" Oct 30 06:25:21.672765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:21.672746 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed23fda-b84c-449c-8b28-3a73be116291" path="/var/lib/kubelet/pods/fed23fda-b84c-449c-8b28-3a73be116291/volumes" Oct 30 06:25:23.011359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:23.011322 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 06:25:23.045920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:23.045891 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 06:25:23.672505 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:23.672471 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:25:26.381448 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:26.381402 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:25:26.991141 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:26.991099 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:25:26.991403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:26.991371 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" containerID="cri-o://2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0" gracePeriod=30 Oct 30 06:25:27.620366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620325 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4"] Oct 30 06:25:27.620852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620836 6171 cpu_manager.go:414] "containerMap: removing container" podUID="59f066db-398d-4399-b724-671357a1e3a4" containerName="revoke-keys" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620855 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f066db-398d-4399-b724-671357a1e3a4" containerName="revoke-keys" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620870 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" containerName="payload-pre-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620876 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" containerName="payload-pre-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620887 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620892 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620905 6171 cpu_manager.go:414] "containerMap: removing container" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620910 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" Oct 30 06:25:27.620928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620929 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fed23fda-b84c-449c-8b28-3a73be116291" containerName="maas-api" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620934 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed23fda-b84c-449c-8b28-3a73be116291" containerName="maas-api" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620940 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" containerName="payload-processing" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.620945 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" containerName="payload-processing" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621023 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f066db-398d-4399-b724-671357a1e3a4" containerName="revoke-keys" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621035 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d43ead2-95cf-4ced-8f6b-1605d1b596c3" containerName="payload-pre-processing" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621056 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7322401-c8ba-474a-817a-0558e3b62a1e" containerName="payload-processing" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621068 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4f5b3e-8631-4de5-a254-29e55f219a6b" containerName="payload-pre-processing" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621078 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed23fda-b84c-449c-8b28-3a73be116291" containerName="maas-api" Oct 30 06:25:27.621261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.621084 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ff403f0-2d6f-4e7f-86a5-a6e30af2c91e" containerName="payload-processing" Oct 30 06:25:27.625731 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.625713 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.647323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.647295 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4"] Oct 30 06:25:27.657112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.657092 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:27.657261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.657137 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:25:27.669916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.669890 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbkg9\" (UniqueName: \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.670074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.670007 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.670190 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.670153 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.770975 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.770947 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.771137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.771011 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbkg9\" (UniqueName: \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.771137 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.771089 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.771575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.771555 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.773496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.773476 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.796430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.796407 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbkg9\" (UniqueName: \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\") pod \"maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:27.939249 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:27.939194 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:28.012308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.012268 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" containerID="cri-o://b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214" gracePeriod=25 Oct 30 06:25:28.046933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.046894 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" containerID="cri-o://11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4" gracePeriod=25 Oct 30 06:25:28.240573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.240543 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4"] Oct 30 06:25:28.241669 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:28.241647 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0637087b_39e7_48e1_9e0c_6ba5864c9791.slice/crio-b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b WatchSource:0}: Error finding container b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b: Status 404 returned error can't find the container with id b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b Oct 30 06:25:28.902133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.902039 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" event={"ID":"0637087b-39e7-48e1-9e0c-6ba5864c9791","Type":"ContainerStarted","Data":"f5a7517187eadc2995fbb20806ccfdf5aedd7645d8a8b6bc167e3abdfe46ff36"} Oct 30 06:25:28.902133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.902105 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" event={"ID":"0637087b-39e7-48e1-9e0c-6ba5864c9791","Type":"ContainerStarted","Data":"b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b"} Oct 30 06:25:28.951396 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:28.951347 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" podStartSLOduration=1.951333629 podStartE2EDuration="1.951333629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:28.949332615 +0000 UTC m=+3187.389709808" watchObservedRunningTime="2026-10-30 06:25:28.951333629 +0000 UTC m=+3187.391710820" Oct 30 06:25:29.090464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:29.090435 6171 patch_prober.go:29] interesting pod/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:25:29.090606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:29.090498 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:25:29.910811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:29.910775 6171 generic.go:356] "Generic (PLEG): container finished" podID="0637087b-39e7-48e1-9e0c-6ba5864c9791" containerID="f5a7517187eadc2995fbb20806ccfdf5aedd7645d8a8b6bc167e3abdfe46ff36" exitCode=0 Oct 30 06:25:29.911220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:29.910854 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" event={"ID":"0637087b-39e7-48e1-9e0c-6ba5864c9791","Type":"ContainerDied","Data":"f5a7517187eadc2995fbb20806ccfdf5aedd7645d8a8b6bc167e3abdfe46ff36"} Oct 30 06:25:30.687360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:30.687329 6171 scope.go:122] "RemoveContainer" containerID="585320088a14033bc3c7c6675bc378d31366911c407b847d76e1e4e192107778" Oct 30 06:25:30.697617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:30.697601 6171 scope.go:122] "RemoveContainer" containerID="946270ecf20b3bd3e0d9269dfd1ec4a30bad8f30c9d87c68c7fc26a6fc931d05" Oct 30 06:25:30.707112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:30.707030 6171 scope.go:122] "RemoveContainer" containerID="46bd08728ee1c18d66b2540dfb36bc94ec366e0cfa060ad1c6b325301fae67fd" Oct 30 06:25:31.059848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.059823 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:31.104965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.104938 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\") pod \"0637087b-39e7-48e1-9e0c-6ba5864c9791\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " Oct 30 06:25:31.105164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.105148 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\" (UniqueName: \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\") pod \"0637087b-39e7-48e1-9e0c-6ba5864c9791\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " Oct 30 06:25:31.105262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.105214 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\") pod \"0637087b-39e7-48e1-9e0c-6ba5864c9791\" (UID: \"0637087b-39e7-48e1-9e0c-6ba5864c9791\") " Oct 30 06:25:31.105581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.105544 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca" pod "0637087b-39e7-48e1-9e0c-6ba5864c9791" (UID: "0637087b-39e7-48e1-9e0c-6ba5864c9791"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:31.107614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.107589 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca" pod "0637087b-39e7-48e1-9e0c-6ba5864c9791" (UID: "0637087b-39e7-48e1-9e0c-6ba5864c9791"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:31.107738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.107718 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9" pod "0637087b-39e7-48e1-9e0c-6ba5864c9791" (UID: "0637087b-39e7-48e1-9e0c-6ba5864c9791"). InnerVolumeSpecName "kube-api-access-wbkg9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:31.206181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.206152 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:31.206181 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.206177 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbkg9\" (UniqueName: \"kubernetes.io/projected/0637087b-39e7-48e1-9e0c-6ba5864c9791-kube-api-access-wbkg9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:31.206358 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.206186 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0637087b-39e7-48e1-9e0c-6ba5864c9791-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:31.928412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.928310 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" event={"ID":"0637087b-39e7-48e1-9e0c-6ba5864c9791","Type":"ContainerDied","Data":"b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b"} Oct 30 06:25:31.928412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.928331 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4" Oct 30 06:25:31.928412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:31.928365 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b79953b8c1f97eb2905ca0a4263b8c0bf19162213f62ac63336bca1fc55d690b" Oct 30 06:25:32.246969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.246943 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:25:32.315820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315793 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.315989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315836 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.315989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315883 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.315989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315901 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.315989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315941 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.315989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.315961 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.316279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316014 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.316279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316097 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.316279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316133 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\") pod \"a6386b48-1c8d-4419-801d-a719daf18cc6\" (UID: \"a6386b48-1c8d-4419-801d-a719daf18cc6\") " Oct 30 06:25:32.316819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316772 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.317233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316822 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.317385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.316968 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.317385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.317013 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.317385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.317166 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:32.320075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.319337 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:32.320075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.319557 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.320075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.319635 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:25:32.320075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.319695 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p" pod "a6386b48-1c8d-4419-801d-a719daf18cc6" (UID: "a6386b48-1c8d-4419-801d-a719daf18cc6"). InnerVolumeSpecName "kube-api-access-wrs4p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:32.417870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417847 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.417870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417870 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrs4p\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-kube-api-access-wrs4p\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417880 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a6386b48-1c8d-4419-801d-a719daf18cc6-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417889 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417896 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417904 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417911 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417919 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.418017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.417925 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a6386b48-1c8d-4419-801d-a719daf18cc6-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.431603 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.431579 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" containerID="cri-o://f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d" gracePeriod=45 Oct 30 06:25:32.672821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.672796 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:25:32.720351 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720317 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\" (UniqueName: \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720365 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720394 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720412 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720456 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720513 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\" (UniqueName: \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\") pod \"250b3345-4955-4a10-8afc-337ddd8b8b1e\" (UID: \"250b3345-4955-4a10-8afc-337ddd8b8b1e\") " Oct 30 06:25:32.720742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720670 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.720909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.720876 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.721890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.721273 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.721890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.721565 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.721890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.721752 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.723370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.723343 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "kube-api-access-cv765". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:32.723678 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.723648 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:32.723892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.723872 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm" pod "250b3345-4955-4a10-8afc-337ddd8b8b1e" (UID: "250b3345-4955-4a10-8afc-337ddd8b8b1e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:25:32.822507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.822427 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/250b3345-4955-4a10-8afc-337ddd8b8b1e-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.822507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.822452 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.822507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.822462 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/250b3345-4955-4a10-8afc-337ddd8b8b1e-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.822507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.822475 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cv765\" (UniqueName: \"kubernetes.io/projected/250b3345-4955-4a10-8afc-337ddd8b8b1e-kube-api-access-cv765\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:32.937589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.937556 6171 generic.go:356] "Generic (PLEG): container finished" podID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerID="f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d" exitCode=0 Oct 30 06:25:32.937799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.937612 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" Oct 30 06:25:32.937799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.937636 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" event={"ID":"250b3345-4955-4a10-8afc-337ddd8b8b1e","Type":"ContainerDied","Data":"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d"} Oct 30 06:25:32.937799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.937689 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4" event={"ID":"250b3345-4955-4a10-8afc-337ddd8b8b1e","Type":"ContainerDied","Data":"573efdb68e51118c73513108214d24f115a2415783a254dc62c3e2148ff07e6b"} Oct 30 06:25:32.937799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.937710 6171 scope.go:122] "RemoveContainer" containerID="f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d" Oct 30 06:25:32.939829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.939803 6171 generic.go:356] "Generic (PLEG): container finished" podID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerID="2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0" exitCode=0 Oct 30 06:25:32.939950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.939910 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" Oct 30 06:25:32.940017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.939946 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerDied","Data":"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0"} Oct 30 06:25:32.940017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.939995 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8" event={"ID":"a6386b48-1c8d-4419-801d-a719daf18cc6","Type":"ContainerDied","Data":"a9379260876614ff4f68fcdc0bac5df11e94cae22ab5d2da6b558b8b3ff4d59b"} Oct 30 06:25:32.949598 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.949548 6171 scope.go:122] "RemoveContainer" containerID="f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d" Oct 30 06:25:32.949874 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:32.949851 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d\": container with ID starting with f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d not found: ID does not exist" containerID="f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d" Oct 30 06:25:32.949972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.949884 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d"} err="failed to get container status \"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d\": rpc error: code = NotFound desc = could not find container \"f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d\": container with ID starting with f4fb1445c327a16c953f389a043b641bcd5ba8ff947e5d9dfbfa73b03543a61d not found: ID does not exist" Oct 30 06:25:32.949972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.949906 6171 scope.go:122] "RemoveContainer" containerID="2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0" Oct 30 06:25:32.962349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.962327 6171 scope.go:122] "RemoveContainer" containerID="8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f" Oct 30 06:25:32.971569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.971545 6171 scope.go:122] "RemoveContainer" containerID="2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0" Oct 30 06:25:32.971814 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:32.971794 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0\": container with ID starting with 2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0 not found: ID does not exist" containerID="2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0" Oct 30 06:25:32.971872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.971824 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0"} err="failed to get container status \"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0\": rpc error: code = NotFound desc = could not find container \"2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0\": container with ID starting with 2c15ec9315f323dbae56e3e15ab9ef4c681618836feb701b8d34c4b166fd84e0 not found: ID does not exist" Oct 30 06:25:32.971872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.971839 6171 scope.go:122] "RemoveContainer" containerID="8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f" Oct 30 06:25:32.972099 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:32.972079 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f\": container with ID starting with 8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f not found: ID does not exist" containerID="8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f" Oct 30 06:25:32.972163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.972109 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f"} err="failed to get container status \"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f\": rpc error: code = NotFound desc = could not find container \"8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f\": container with ID starting with 8503ed3702904df540532400db1fc7799452a9fb40cb1aa79dcaab018585972f not found: ID does not exist" Oct 30 06:25:32.997213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:32.997190 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:25:33.006420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:33.004408 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-1685dd-openshift-default-79d84f767-czwd8"] Oct 30 06:25:33.043187 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:33.043164 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 06:25:33.058430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:33.058406 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-e2e3d0/premium-w4-e2e3d0-kserve-768877854c-hs8k4"] Oct 30 06:25:33.670474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:33.670394 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" path="/var/lib/kubelet/pods/250b3345-4955-4a10-8afc-337ddd8b8b1e/volumes" Oct 30 06:25:33.670850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:33.670805 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" path="/var/lib/kubelet/pods/a6386b48-1c8d-4419-801d-a719daf18cc6/volumes" Oct 30 06:25:34.023156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023128 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht"] Oct 30 06:25:34.023563 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023551 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.023605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023564 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.023605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023581 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.023605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023586 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.023605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023603 6171 cpu_manager.go:414] "containerMap: removing container" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023608 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023623 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0637087b-39e7-48e1-9e0c-6ba5864c9791" containerName="revoke-keys" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023628 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0637087b-39e7-48e1-9e0c-6ba5864c9791" containerName="revoke-keys" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023696 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="250b3345-4955-4a10-8afc-337ddd8b8b1e" containerName="main" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023705 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0637087b-39e7-48e1-9e0c-6ba5864c9791" containerName="revoke-keys" Oct 30 06:25:34.023737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.023714 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.028360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.028334 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.047214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.047196 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:34.047365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.047348 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht"] Oct 30 06:25:34.058516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.058493 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:25:34.061662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.061643 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm"] Oct 30 06:25:34.062180 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.062166 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6386b48-1c8d-4419-801d-a719daf18cc6" containerName="istio-proxy" Oct 30 06:25:34.065426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.065410 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.078006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.077944 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm"] Oct 30 06:25:34.136763 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.136733 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.136881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.136768 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.136881 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.136805 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wmcb\" (UniqueName: \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.136960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.136937 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.136994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.136982 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5vpw\" (UniqueName: \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.137077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.137036 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.238293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238259 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.238293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238295 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.238516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238322 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wmcb\" (UniqueName: \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.238516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238371 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.238516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238401 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5vpw\" (UniqueName: \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.238516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.238427 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.239207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.239183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.239332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.239183 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.240921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.240899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.241015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.240921 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.263742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.263712 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wmcb\" (UniqueName: \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\") pod \"maas-api-revoke-sub-3c520a6f-2lrht\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.263845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.263830 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5vpw\" (UniqueName: \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\") pod \"maas-api-revoke-sub-2878804e-dgzbm\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.342825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.342750 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:34.380155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.380124 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:34.648224 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.648193 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht"] Oct 30 06:25:34.648622 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:34.648599 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649d06a2_6652_47f9_8443_8f83193c55e0.slice/crio-f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52 WatchSource:0}: Error finding container f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52: Status 404 returned error can't find the container with id f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52 Oct 30 06:25:34.716950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.716923 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm"] Oct 30 06:25:34.717614 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:34.717589 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eec602c_90c1_48e0_9c27_8e91f7876591.slice/crio-bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc WatchSource:0}: Error finding container bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc: Status 404 returned error can't find the container with id bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc Oct 30 06:25:34.958378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.958345 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" event={"ID":"649d06a2-6652-47f9-8443-8f83193c55e0","Type":"ContainerStarted","Data":"9488ff406f9e3a795f1ca3882267f3c4010ea154409fb8beaff3b9d4b29365d7"} Oct 30 06:25:34.958496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.958407 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" event={"ID":"649d06a2-6652-47f9-8443-8f83193c55e0","Type":"ContainerStarted","Data":"f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52"} Oct 30 06:25:34.959615 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:34.959591 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" event={"ID":"4eec602c-90c1-48e0-9c27-8e91f7876591","Type":"ContainerStarted","Data":"bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc"} Oct 30 06:25:35.007158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:35.007115 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" podStartSLOduration=2.007102113 podStartE2EDuration="2.007102113s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:35.004285356 +0000 UTC m=+3193.444662561" watchObservedRunningTime="2026-10-30 06:25:35.007102113 +0000 UTC m=+3193.447479320" Oct 30 06:25:35.968129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:35.968096 6171 generic.go:356] "Generic (PLEG): container finished" podID="649d06a2-6652-47f9-8443-8f83193c55e0" containerID="9488ff406f9e3a795f1ca3882267f3c4010ea154409fb8beaff3b9d4b29365d7" exitCode=0 Oct 30 06:25:35.968599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:35.968184 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" event={"ID":"649d06a2-6652-47f9-8443-8f83193c55e0","Type":"ContainerDied","Data":"9488ff406f9e3a795f1ca3882267f3c4010ea154409fb8beaff3b9d4b29365d7"} Oct 30 06:25:35.969815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:35.969793 6171 generic.go:356] "Generic (PLEG): container finished" podID="4eec602c-90c1-48e0-9c27-8e91f7876591" containerID="dff3eb970f32a788e24e92f85f80561309775570b9d8f1d7c86fc1afc837aa84" exitCode=0 Oct 30 06:25:35.969946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:35.969868 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" event={"ID":"4eec602c-90c1-48e0-9c27-8e91f7876591","Type":"ContainerDied","Data":"dff3eb970f32a788e24e92f85f80561309775570b9d8f1d7c86fc1afc837aa84"} Oct 30 06:25:37.109057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.109019 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:37.142163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.142140 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:37.167742 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.167712 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\" (UniqueName: \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\") pod \"4eec602c-90c1-48e0-9c27-8e91f7876591\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " Oct 30 06:25:37.167873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.167764 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\" (UniqueName: \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\") pod \"649d06a2-6652-47f9-8443-8f83193c55e0\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " Oct 30 06:25:37.167873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.167814 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\") pod \"649d06a2-6652-47f9-8443-8f83193c55e0\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " Oct 30 06:25:37.168003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.167871 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\") pod \"4eec602c-90c1-48e0-9c27-8e91f7876591\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " Oct 30 06:25:37.168003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.167988 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\") pod \"649d06a2-6652-47f9-8443-8f83193c55e0\" (UID: \"649d06a2-6652-47f9-8443-8f83193c55e0\") " Oct 30 06:25:37.168260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.168192 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\") pod \"4eec602c-90c1-48e0-9c27-8e91f7876591\" (UID: \"4eec602c-90c1-48e0-9c27-8e91f7876591\") " Oct 30 06:25:37.168412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.168383 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca" pod "649d06a2-6652-47f9-8443-8f83193c55e0" (UID: "649d06a2-6652-47f9-8443-8f83193c55e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:37.168893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.168867 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca" pod "4eec602c-90c1-48e0-9c27-8e91f7876591" (UID: "4eec602c-90c1-48e0-9c27-8e91f7876591"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:37.168995 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.168904 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.170154 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.170127 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca" pod "4eec602c-90c1-48e0-9c27-8e91f7876591" (UID: "4eec602c-90c1-48e0-9c27-8e91f7876591"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:37.170243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.170165 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw" pod "4eec602c-90c1-48e0-9c27-8e91f7876591" (UID: "4eec602c-90c1-48e0-9c27-8e91f7876591"). InnerVolumeSpecName "kube-api-access-n5vpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:37.170558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.170537 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca" pod "649d06a2-6652-47f9-8443-8f83193c55e0" (UID: "649d06a2-6652-47f9-8443-8f83193c55e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:37.170852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.170835 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb" pod "649d06a2-6652-47f9-8443-8f83193c55e0" (UID: "649d06a2-6652-47f9-8443-8f83193c55e0"). InnerVolumeSpecName "kube-api-access-7wmcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:37.201297 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.201267 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:37.201409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.201274 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:37.269938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.269881 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5vpw\" (UniqueName: \"kubernetes.io/projected/4eec602c-90c1-48e0-9c27-8e91f7876591-kube-api-access-n5vpw\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.269938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.269905 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wmcb\" (UniqueName: \"kubernetes.io/projected/649d06a2-6652-47f9-8443-8f83193c55e0-kube-api-access-7wmcb\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.269938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.269914 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649d06a2-6652-47f9-8443-8f83193c55e0-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.269938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.269923 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.269938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.269932 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4eec602c-90c1-48e0-9c27-8e91f7876591-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:37.985926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.985895 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" Oct 30 06:25:37.986155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.985980 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm" event={"ID":"4eec602c-90c1-48e0-9c27-8e91f7876591","Type":"ContainerDied","Data":"bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc"} Oct 30 06:25:37.986155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.986089 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd9c3e5d208a06aecaf46186404156dfd960d79021f8b07bdf1d1647a42069cc" Oct 30 06:25:37.988082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.987774 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" event={"ID":"649d06a2-6652-47f9-8443-8f83193c55e0","Type":"ContainerDied","Data":"f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52"} Oct 30 06:25:37.988082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.987815 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2b754a2211d81e8da004072a3cbb958ebccc6d11d28232c3705b0292626aa52" Oct 30 06:25:37.988082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:37.987838 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht" Oct 30 06:25:41.159985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.159948 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:25:41.160593 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.160209 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" podUID="08c677cf-faba-4522-bb38-4e6ded8f141b" containerName="maas-api" containerID="cri-o://d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2" gracePeriod=30 Oct 30 06:25:41.429408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.429380 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:25:41.510391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.510358 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\") pod \"08c677cf-faba-4522-bb38-4e6ded8f141b\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " Oct 30 06:25:41.510558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.510456 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\" (UniqueName: \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\") pod \"08c677cf-faba-4522-bb38-4e6ded8f141b\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " Oct 30 06:25:41.510558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.510512 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\") pod \"08c677cf-faba-4522-bb38-4e6ded8f141b\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " Oct 30 06:25:41.510701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.510584 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\") pod \"08c677cf-faba-4522-bb38-4e6ded8f141b\" (UID: \"08c677cf-faba-4522-bb38-4e6ded8f141b\") " Oct 30 06:25:41.510790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.510757 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca" pod "08c677cf-faba-4522-bb38-4e6ded8f141b" (UID: "08c677cf-faba-4522-bb38-4e6ded8f141b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:41.511147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.511064 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08c677cf-faba-4522-bb38-4e6ded8f141b-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.512704 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.512678 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls" pod "08c677cf-faba-4522-bb38-4e6ded8f141b" (UID: "08c677cf-faba-4522-bb38-4e6ded8f141b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:41.512811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.512732 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs" pod "08c677cf-faba-4522-bb38-4e6ded8f141b" (UID: "08c677cf-faba-4522-bb38-4e6ded8f141b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:41.512811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.512802 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v" pod "08c677cf-faba-4522-bb38-4e6ded8f141b" (UID: "08c677cf-faba-4522-bb38-4e6ded8f141b"). InnerVolumeSpecName "kube-api-access-2lb9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:41.612257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.612230 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lb9v\" (UniqueName: \"kubernetes.io/projected/08c677cf-faba-4522-bb38-4e6ded8f141b-kube-api-access-2lb9v\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.612257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.612258 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:41.612442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:41.612271 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08c677cf-faba-4522-bb38-4e6ded8f141b-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:42.033718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.033677 6171 generic.go:356] "Generic (PLEG): container finished" podID="08c677cf-faba-4522-bb38-4e6ded8f141b" containerID="d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2" exitCode=0 Oct 30 06:25:42.033909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.033762 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" event={"ID":"08c677cf-faba-4522-bb38-4e6ded8f141b","Type":"ContainerDied","Data":"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2"} Oct 30 06:25:42.033909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.033831 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" event={"ID":"08c677cf-faba-4522-bb38-4e6ded8f141b","Type":"ContainerDied","Data":"5d6c8a9aaed945b82c7622319192dab17fc617232700041177bea625f310c959"} Oct 30 06:25:42.033909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.033852 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5" Oct 30 06:25:42.034135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.033857 6171 scope.go:122] "RemoveContainer" containerID="d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2" Oct 30 06:25:42.044716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.044693 6171 scope.go:122] "RemoveContainer" containerID="d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2" Oct 30 06:25:42.045038 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:42.045019 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2\": container with ID starting with d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2 not found: ID does not exist" containerID="d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2" Oct 30 06:25:42.045127 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.045056 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2"} err="failed to get container status \"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2\": rpc error: code = NotFound desc = could not find container \"d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2\": container with ID starting with d53147db9dd9194ed0857c2182021c7dd82fa970a31822917ffa55f2552d6bf2 not found: ID does not exist" Oct 30 06:25:42.094939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.094909 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:25:42.106914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.106887 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-630fdf-844986f489-5vwn5"] Oct 30 06:25:42.109569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.109548 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4"] Oct 30 06:25:42.111859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:42.111838 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-630fdf-c58b4"] Oct 30 06:25:43.670934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:43.670892 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0637087b-39e7-48e1-9e0c-6ba5864c9791" path="/var/lib/kubelet/pods/0637087b-39e7-48e1-9e0c-6ba5864c9791/volumes" Oct 30 06:25:43.671604 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:43.671582 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08c677cf-faba-4522-bb38-4e6ded8f141b" path="/var/lib/kubelet/pods/08c677cf-faba-4522-bb38-4e6ded8f141b/volumes" Oct 30 06:25:45.865112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:45.865080 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:25:47.201072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:47.201008 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:47.201537 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:47.201178 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:25:48.293983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.293961 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 06:25:48.323117 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.323093 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 06:25:48.376783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.376757 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\") pod \"e9689235-35dd-4ef4-a797-8c0e7c123edd\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " Oct 30 06:25:48.376925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.376817 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\" (UniqueName: \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\") pod \"e9689235-35dd-4ef4-a797-8c0e7c123edd\" (UID: \"e9689235-35dd-4ef4-a797-8c0e7c123edd\") " Oct 30 06:25:48.376925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.376850 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\") pod \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " Oct 30 06:25:48.377106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.377084 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\" (UniqueName: \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\") pod \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\" (UID: \"5b3ad0c3-ff77-4811-b3be-e55e91ed545d\") " Oct 30 06:25:48.377218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.377190 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume" pod "e9689235-35dd-4ef4-a797-8c0e7c123edd" (UID: "e9689235-35dd-4ef4-a797-8c0e7c123edd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:48.377342 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.377191 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume" pod "5b3ad0c3-ff77-4811-b3be-e55e91ed545d" (UID: "5b3ad0c3-ff77-4811-b3be-e55e91ed545d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:48.377503 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.377486 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e9689235-35dd-4ef4-a797-8c0e7c123edd-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.377583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.377504 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.379149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.379131 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4" pod "e9689235-35dd-4ef4-a797-8c0e7c123edd" (UID: "e9689235-35dd-4ef4-a797-8c0e7c123edd"). InnerVolumeSpecName "kube-api-access-bc2v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:48.379232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.379156 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn" pod "5b3ad0c3-ff77-4811-b3be-e55e91ed545d" (UID: "5b3ad0c3-ff77-4811-b3be-e55e91ed545d"). InnerVolumeSpecName "kube-api-access-k9wnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:48.408504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.408471 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj"] Oct 30 06:25:48.411647 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.411609 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-e2e3d0-z7xhj"] Oct 30 06:25:48.478020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.477988 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9wnn\" (UniqueName: \"kubernetes.io/projected/5b3ad0c3-ff77-4811-b3be-e55e91ed545d-kube-api-access-k9wnn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:48.478020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:48.478012 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc2v4\" (UniqueName: \"kubernetes.io/projected/e9689235-35dd-4ef4-a797-8c0e7c123edd-kube-api-access-bc2v4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:49.088888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.088855 6171 generic.go:356] "Generic (PLEG): container finished" podID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerID="11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4" exitCode=0 Oct 30 06:25:49.089093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.088921 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" Oct 30 06:25:49.089093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.088939 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" event={"ID":"5b3ad0c3-ff77-4811-b3be-e55e91ed545d","Type":"ContainerDied","Data":"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4"} Oct 30 06:25:49.089093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.088995 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8" event={"ID":"5b3ad0c3-ff77-4811-b3be-e55e91ed545d","Type":"ContainerDied","Data":"b0dda93e09b0c46484a1a7dba04ecab2419b805d1a72eeaa83fad0cb4c613e9e"} Oct 30 06:25:49.089093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.089025 6171 scope.go:122] "RemoveContainer" containerID="11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4" Oct 30 06:25:49.090542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.090515 6171 generic.go:356] "Generic (PLEG): container finished" podID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerID="b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214" exitCode=0 Oct 30 06:25:49.090672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.090568 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" event={"ID":"e9689235-35dd-4ef4-a797-8c0e7c123edd","Type":"ContainerDied","Data":"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214"} Oct 30 06:25:49.090672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.090615 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" event={"ID":"e9689235-35dd-4ef4-a797-8c0e7c123edd","Type":"ContainerDied","Data":"bce56b2866fc43e0b969a3a4e4b04c16e10bff6ca557e72410c7b416fc2c5767"} Oct 30 06:25:49.090672 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.090581 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h" Oct 30 06:25:49.100471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.100438 6171 scope.go:122] "RemoveContainer" containerID="11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4" Oct 30 06:25:49.100750 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:49.100732 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4\": container with ID starting with 11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4 not found: ID does not exist" containerID="11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4" Oct 30 06:25:49.100815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.100757 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4"} err="failed to get container status \"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4\": rpc error: code = NotFound desc = could not find container \"11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4\": container with ID starting with 11dbc021703213e505e07231efb3c3dca4be34c74477cb79b850803948cc53c4 not found: ID does not exist" Oct 30 06:25:49.100815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.100774 6171 scope.go:122] "RemoveContainer" containerID="b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214" Oct 30 06:25:49.109667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.109651 6171 scope.go:122] "RemoveContainer" containerID="b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214" Oct 30 06:25:49.109918 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:49.109899 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214\": container with ID starting with b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214 not found: ID does not exist" containerID="b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214" Oct 30 06:25:49.109967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.109925 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214"} err="failed to get container status \"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214\": rpc error: code = NotFound desc = could not find container \"b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214\": container with ID starting with b4bffec97a40b4850a402bac9e486de6e9cee6329f28e32377be4f23b08d0214 not found: ID does not exist" Oct 30 06:25:49.155156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.155135 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 06:25:49.168946 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.168924 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-e2e3d0-6bd4b555b5-lm86h"] Oct 30 06:25:49.200732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.200710 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 06:25:49.211275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.211256 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-e2e3d0-6dcf75c64c-swrb8"] Oct 30 06:25:49.669920 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.669890 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f066db-398d-4399-b724-671357a1e3a4" path="/var/lib/kubelet/pods/59f066db-398d-4399-b724-671357a1e3a4/volumes" Oct 30 06:25:49.670317 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.670265 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" path="/var/lib/kubelet/pods/5b3ad0c3-ff77-4811-b3be-e55e91ed545d/volumes" Oct 30 06:25:49.670579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:49.670567 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" path="/var/lib/kubelet/pods/e9689235-35dd-4ef4-a797-8c0e7c123edd/volumes" Oct 30 06:25:50.726414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726383 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg"] Oct 30 06:25:50.726833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726820 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" Oct 30 06:25:50.726833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726833 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726848 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726853 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726867 6171 cpu_manager.go:414] "containerMap: removing container" podUID="649d06a2-6652-47f9-8443-8f83193c55e0" containerName="revoke-keys" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726872 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649d06a2-6652-47f9-8443-8f83193c55e0" containerName="revoke-keys" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726885 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4eec602c-90c1-48e0-9c27-8e91f7876591" containerName="revoke-keys" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726889 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4eec602c-90c1-48e0-9c27-8e91f7876591" containerName="revoke-keys" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726898 6171 cpu_manager.go:414] "containerMap: removing container" podUID="08c677cf-faba-4522-bb38-4e6ded8f141b" containerName="maas-api" Oct 30 06:25:50.726937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726903 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08c677cf-faba-4522-bb38-4e6ded8f141b" containerName="maas-api" Oct 30 06:25:50.727275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726963 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b3ad0c3-ff77-4811-b3be-e55e91ed545d" containerName="payload-pre-processing" Oct 30 06:25:50.727275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726973 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9689235-35dd-4ef4-a797-8c0e7c123edd" containerName="payload-processing" Oct 30 06:25:50.727275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726980 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="649d06a2-6652-47f9-8443-8f83193c55e0" containerName="revoke-keys" Oct 30 06:25:50.727275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726987 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4eec602c-90c1-48e0-9c27-8e91f7876591" containerName="revoke-keys" Oct 30 06:25:50.727275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.726992 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="08c677cf-faba-4522-bb38-4e6ded8f141b" containerName="maas-api" Oct 30 06:25:50.731737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.731722 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.753625 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.753593 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg"] Oct 30 06:25:50.764280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.764255 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:25:50.764280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.764262 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:25:50.764465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.764329 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:25:50.764465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.764351 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:25:50.800607 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.800579 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.800710 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.800657 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.800757 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.800736 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67wqr\" (UniqueName: \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.901500 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.901477 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67wqr\" (UniqueName: \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.901610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.901528 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.901610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.901574 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.902251 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.902227 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.904000 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.903983 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:50.931237 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:50.931212 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67wqr\" (UniqueName: \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:51.044939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:51.044876 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:51.348497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:51.348465 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg"] Oct 30 06:25:51.349102 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:25:51.349076 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11b93af9_0a84_4292_a2e8_1837a29bbfb3.slice/crio-eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f WatchSource:0}: Error finding container eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f: Status 404 returned error can't find the container with id eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f Oct 30 06:25:52.121263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:52.121226 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" event={"ID":"11b93af9-0a84-4292-a2e8-1837a29bbfb3","Type":"ContainerStarted","Data":"5347dd7ba6391055752a23127fde2c2a8b728eced2f05aa919808c2f7acd00cd"} Oct 30 06:25:52.121633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:52.121285 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" event={"ID":"11b93af9-0a84-4292-a2e8-1837a29bbfb3","Type":"ContainerStarted","Data":"eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f"} Oct 30 06:25:52.167943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:52.167900 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" podStartSLOduration=2.167887447 podStartE2EDuration="2.167887447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:52.165247756 +0000 UTC m=+3210.605624947" watchObservedRunningTime="2026-10-30 06:25:52.167887447 +0000 UTC m=+3210.608264639" Oct 30 06:25:53.130282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:53.130248 6171 generic.go:356] "Generic (PLEG): container finished" podID="11b93af9-0a84-4292-a2e8-1837a29bbfb3" containerID="5347dd7ba6391055752a23127fde2c2a8b728eced2f05aa919808c2f7acd00cd" exitCode=0 Oct 30 06:25:53.130692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:53.130339 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" event={"ID":"11b93af9-0a84-4292-a2e8-1837a29bbfb3","Type":"ContainerDied","Data":"5347dd7ba6391055752a23127fde2c2a8b728eced2f05aa919808c2f7acd00cd"} Oct 30 06:25:54.049256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.049223 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj"] Oct 30 06:25:54.062659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.062634 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4a9918b1-gnddj"] Oct 30 06:25:54.298201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.298177 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:54.337429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.337351 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\" (UniqueName: \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\") pod \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " Oct 30 06:25:54.337429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.337411 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\") pod \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " Oct 30 06:25:54.337631 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.337454 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\") pod \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\" (UID: \"11b93af9-0a84-4292-a2e8-1837a29bbfb3\") " Oct 30 06:25:54.337841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.337816 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca" pod "11b93af9-0a84-4292-a2e8-1837a29bbfb3" (UID: "11b93af9-0a84-4292-a2e8-1837a29bbfb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:25:54.339655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.339635 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr" pod "11b93af9-0a84-4292-a2e8-1837a29bbfb3" (UID: "11b93af9-0a84-4292-a2e8-1837a29bbfb3"). InnerVolumeSpecName "kube-api-access-67wqr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:25:54.339756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.339701 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca" pod "11b93af9-0a84-4292-a2e8-1837a29bbfb3" (UID: "11b93af9-0a84-4292-a2e8-1837a29bbfb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:25:54.439275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.439239 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67wqr\" (UniqueName: \"kubernetes.io/projected/11b93af9-0a84-4292-a2e8-1837a29bbfb3-kube-api-access-67wqr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.439275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.439269 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:54.439457 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:54.439282 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11b93af9-0a84-4292-a2e8-1837a29bbfb3-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:25:55.148293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:55.148267 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" Oct 30 06:25:55.148472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:55.148290 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg" event={"ID":"11b93af9-0a84-4292-a2e8-1837a29bbfb3","Type":"ContainerDied","Data":"eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f"} Oct 30 06:25:55.148472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:55.148335 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eca57a32a19ff4369a0ce320a72d94b0a0dc9986282bbedf806e70be93e14e5f" Oct 30 06:25:55.670207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:25:55.670173 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e01b3c7-31bb-429c-a80a-f292b4ce86c2" path="/var/lib/kubelet/pods/7e01b3c7-31bb-429c-a80a-f292b4ce86c2/volumes" Oct 30 06:25:56.378861 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:25:56.378821 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:26:01.179712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:01.179674 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:26:01.546235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:01.546199 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:26:04.035192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:04.035157 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682"] Oct 30 06:26:04.049453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:04.049423 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0eb6584f-29682"] Oct 30 06:26:05.670307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:05.670276 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="481f57f4-862d-4cd1-85ed-7a78c6994650" path="/var/lib/kubelet/pods/481f57f4-862d-4cd1-85ed-7a78c6994650/volumes" Oct 30 06:26:06.041614 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:06.041577 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg"] Oct 30 06:26:06.057135 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:06.057108 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-25d881-zv5bg"] Oct 30 06:26:06.180017 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:06.179977 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" containerID="cri-o://519757b1362b2ddf68dfce56d489679305596e0951b2956ba10073b4383a2117" gracePeriod=25 Oct 30 06:26:07.671066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:07.670999 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11b93af9-0a84-4292-a2e8-1837a29bbfb3" path="/var/lib/kubelet/pods/11b93af9-0a84-4292-a2e8-1837a29bbfb3/volumes" Oct 30 06:26:08.669815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:08.669786 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:26:10.950838 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:10.950799 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:13.649028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:13.648995 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:26:13.649464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:13.649247 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" containerID="cri-o://a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4" gracePeriod=30 Oct 30 06:26:14.235323 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.235285 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85"] Oct 30 06:26:14.235747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.235734 6171 cpu_manager.go:414] "containerMap: removing container" podUID="11b93af9-0a84-4292-a2e8-1837a29bbfb3" containerName="revoke-keys" Oct 30 06:26:14.235792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.235748 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11b93af9-0a84-4292-a2e8-1837a29bbfb3" containerName="revoke-keys" Oct 30 06:26:14.235833 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.235824 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="11b93af9-0a84-4292-a2e8-1837a29bbfb3" containerName="revoke-keys" Oct 30 06:26:14.240673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.240653 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.267579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.267552 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85"] Oct 30 06:26:14.277450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.277424 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:14.277576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.277459 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:14.277576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.277464 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:14.277576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.277544 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:26:14.338655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.338630 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.338790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.338667 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzt7s\" (UniqueName: \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.338790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.338700 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.439507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.439479 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.439630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.439521 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzt7s\" (UniqueName: \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.439630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.439551 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.440173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.440152 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.441949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.441924 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.465144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.465124 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzt7s\" (UniqueName: \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.554001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.553940 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:14.858561 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:14.858537 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85"] Oct 30 06:26:14.860084 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:26:14.860033 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77cc2f1_f992_4fe7_8d03_b64444a8e307.slice/crio-8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030 WatchSource:0}: Error finding container 8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030: Status 404 returned error can't find the container with id 8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030 Oct 30 06:26:15.302549 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:15.302511 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" event={"ID":"e77cc2f1-f992-4fe7-8d03-b64444a8e307","Type":"ContainerStarted","Data":"ed478e2e01b521ed049f72962d5f46d56719711a6fb7e8ed84e4735c14f9906c"} Oct 30 06:26:15.302719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:15.302569 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" event={"ID":"e77cc2f1-f992-4fe7-8d03-b64444a8e307","Type":"ContainerStarted","Data":"8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030"} Oct 30 06:26:15.354534 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:15.354486 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" podStartSLOduration=1.354474321 podStartE2EDuration="1.354474321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:15.35125452 +0000 UTC m=+3233.791631712" watchObservedRunningTime="2026-10-30 06:26:15.354474321 +0000 UTC m=+3233.794851511" Oct 30 06:26:16.311790 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:16.311757 6171 generic.go:356] "Generic (PLEG): container finished" podID="e77cc2f1-f992-4fe7-8d03-b64444a8e307" containerID="ed478e2e01b521ed049f72962d5f46d56719711a6fb7e8ed84e4735c14f9906c" exitCode=0 Oct 30 06:26:16.312244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:16.311841 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" event={"ID":"e77cc2f1-f992-4fe7-8d03-b64444a8e307","Type":"ContainerDied","Data":"ed478e2e01b521ed049f72962d5f46d56719711a6fb7e8ed84e4735c14f9906c"} Oct 30 06:26:17.453989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.453967 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:17.573006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.572919 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\") pod \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " Oct 30 06:26:17.573006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.572985 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\") pod \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " Oct 30 06:26:17.573298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.573008 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\" (UniqueName: \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\") pod \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\" (UID: \"e77cc2f1-f992-4fe7-8d03-b64444a8e307\") " Oct 30 06:26:17.573371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.573349 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca" pod "e77cc2f1-f992-4fe7-8d03-b64444a8e307" (UID: "e77cc2f1-f992-4fe7-8d03-b64444a8e307"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:17.573528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.573506 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:17.575226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.575198 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca" pod "e77cc2f1-f992-4fe7-8d03-b64444a8e307" (UID: "e77cc2f1-f992-4fe7-8d03-b64444a8e307"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:17.575371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.575219 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s" pod "e77cc2f1-f992-4fe7-8d03-b64444a8e307" (UID: "e77cc2f1-f992-4fe7-8d03-b64444a8e307"). InnerVolumeSpecName "kube-api-access-jzt7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:17.674081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.674030 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e77cc2f1-f992-4fe7-8d03-b64444a8e307-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:17.674081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:17.674075 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzt7s\" (UniqueName: \"kubernetes.io/projected/e77cc2f1-f992-4fe7-8d03-b64444a8e307-kube-api-access-jzt7s\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:18.335960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:18.335926 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" Oct 30 06:26:18.336166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:18.335965 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85" event={"ID":"e77cc2f1-f992-4fe7-8d03-b64444a8e307","Type":"ContainerDied","Data":"8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030"} Oct 30 06:26:18.336166 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:18.336034 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8164edb9895e76a6d9a909067b1d7c1d0541b39912d14991d7a8b10c8b59e030" Oct 30 06:26:18.902529 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:18.902505 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:26:19.086876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.086846 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.086896 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.086940 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.086997 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087078 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087112 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087147 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087189 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:19.087373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087316 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087357 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\") pod \"5ed69623-150a-4bc7-864b-b1bd7e822605\" (UID: \"5ed69623-150a-4bc7-864b-b1bd7e822605\") " Oct 30 06:26:19.087659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087499 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:19.087659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087479 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:19.087659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087532 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:19.087800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087691 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:19.088007 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.087987 6171 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-data\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.088077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.088015 6171 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-credential-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.088077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.088030 6171 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.088077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.088072 6171 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-workload-socket\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.088228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.088088 6171 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5ed69623-150a-4bc7-864b-b1bd7e822605-istiod-ca-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.089370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.089344 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:26:19.089473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.089436 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:19.089792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.089778 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:19.090076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.090029 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j" pod "5ed69623-150a-4bc7-864b-b1bd7e822605" (UID: "5ed69623-150a-4bc7-864b-b1bd7e822605"). InnerVolumeSpecName "kube-api-access-f498j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:19.188807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.188780 6171 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-token\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.188917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.188808 6171 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-podinfo\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.188917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.188824 6171 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5ed69623-150a-4bc7-864b-b1bd7e822605-istio-envoy\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.188917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.188836 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f498j\" (UniqueName: \"kubernetes.io/projected/5ed69623-150a-4bc7-864b-b1bd7e822605-kube-api-access-f498j\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:19.345138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.345025 6171 generic.go:356] "Generic (PLEG): container finished" podID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerID="a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4" exitCode=0 Oct 30 06:26:19.345138 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.345113 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" Oct 30 06:26:19.345347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.345113 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" event={"ID":"5ed69623-150a-4bc7-864b-b1bd7e822605","Type":"ContainerDied","Data":"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4"} Oct 30 06:26:19.345347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.345249 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646" event={"ID":"5ed69623-150a-4bc7-864b-b1bd7e822605","Type":"ContainerDied","Data":"40db64ce3ae393663982f9bdf807f8bbd5113a8d76f2aae153d9d3cefeb7bacb"} Oct 30 06:26:19.345347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.345273 6171 scope.go:122] "RemoveContainer" containerID="a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4" Oct 30 06:26:19.356788 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.356772 6171 scope.go:122] "RemoveContainer" containerID="a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4" Oct 30 06:26:19.357065 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:19.357028 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4\": container with ID starting with a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4 not found: ID does not exist" containerID="a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4" Oct 30 06:26:19.357150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.357080 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4"} err="failed to get container status \"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4\": rpc error: code = NotFound desc = could not find container \"a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4\": container with ID starting with a13c490c8da1f9d3b5d867c88a87120702f2e87f64fd77fc056532ddc8961ce4 not found: ID does not exist" Oct 30 06:26:19.403891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.403865 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:26:19.417909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.417884 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-25d881-openshift-default-6dc756c6b7-hd646"] Oct 30 06:26:19.672242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:19.672163 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" path="/var/lib/kubelet/pods/5ed69623-150a-4bc7-864b-b1bd7e822605/volumes" Oct 30 06:26:20.950931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:20.950892 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:24.661420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:24.661384 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:26:26.046446 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.046409 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf"] Oct 30 06:26:26.059956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.059918 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51398201-cwfkf"] Oct 30 06:26:26.139152 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.139122 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85"] Oct 30 06:26:26.141493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.141473 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-f6302c-xbx85"] Oct 30 06:26:26.387263 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:26.386232 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:26:26.399733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.399698 6171 generic.go:356] "Generic (PLEG): container finished" podID="6da28380-2c55-46e6-9d73-4656683abb32" containerID="519757b1362b2ddf68dfce56d489679305596e0951b2956ba10073b4383a2117" exitCode=0 Oct 30 06:26:26.399885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.399767 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" event={"ID":"6da28380-2c55-46e6-9d73-4656683abb32","Type":"ContainerDied","Data":"519757b1362b2ddf68dfce56d489679305596e0951b2956ba10073b4383a2117"} Oct 30 06:26:26.433284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.433262 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:26:26.449978 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.449955 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\") pod \"6da28380-2c55-46e6-9d73-4656683abb32\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " Oct 30 06:26:26.450133 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.449993 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\" (UniqueName: \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\") pod \"6da28380-2c55-46e6-9d73-4656683abb32\" (UID: \"6da28380-2c55-46e6-9d73-4656683abb32\") " Oct 30 06:26:26.450405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.450381 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume" pod "6da28380-2c55-46e6-9d73-4656683abb32" (UID: "6da28380-2c55-46e6-9d73-4656683abb32"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:26.452182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.452158 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4" pod "6da28380-2c55-46e6-9d73-4656683abb32" (UID: "6da28380-2c55-46e6-9d73-4656683abb32"). InnerVolumeSpecName "kube-api-access-dv2k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:26.551261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.551193 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6da28380-2c55-46e6-9d73-4656683abb32-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:26.551261 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:26.551221 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv2k4\" (UniqueName: \"kubernetes.io/projected/6da28380-2c55-46e6-9d73-4656683abb32-kube-api-access-dv2k4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:27.408214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.408178 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" Oct 30 06:26:27.408214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.408202 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz" event={"ID":"6da28380-2c55-46e6-9d73-4656683abb32","Type":"ContainerDied","Data":"0fd7341da5f1929c374a1b354130bb60f1cb6fab9598038008c84ab5f8e9abe1"} Oct 30 06:26:27.408737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.408264 6171 scope.go:122] "RemoveContainer" containerID="519757b1362b2ddf68dfce56d489679305596e0951b2956ba10073b4383a2117" Oct 30 06:26:27.466587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.466564 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:26:27.483661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.483626 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-25d881-7b4cc66599-2gsgz"] Oct 30 06:26:27.670021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.669933 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6da28380-2c55-46e6-9d73-4656683abb32" path="/var/lib/kubelet/pods/6da28380-2c55-46e6-9d73-4656683abb32/volumes" Oct 30 06:26:27.670502 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.670478 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="969fab4d-eea5-4ec9-839c-88a091c2cf2a" path="/var/lib/kubelet/pods/969fab4d-eea5-4ec9-839c-88a091c2cf2a/volumes" Oct 30 06:26:27.670910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:27.670891 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e77cc2f1-f992-4fe7-8d03-b64444a8e307" path="/var/lib/kubelet/pods/e77cc2f1-f992-4fe7-8d03-b64444a8e307/volumes" Oct 30 06:26:28.646973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:28.646938 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:26:29.662334 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:29.662297 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" containerID="cri-o://bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a" gracePeriod=25 Oct 30 06:26:30.788907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:30.788874 6171 scope.go:122] "RemoveContainer" containerID="0d73fe1258fc1b89e961fa36d9ed1a221731d04f8c4e72b8e094583a835a4b2d" Oct 30 06:26:30.799283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:30.799262 6171 scope.go:122] "RemoveContainer" containerID="afad4057133708057278d787949e3f87c046346ccd789334cf66e10589271d72" Oct 30 06:26:30.808605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:30.808585 6171 scope.go:122] "RemoveContainer" containerID="189c968c35401fbc452e73b26657f8c962a4a013c098a7ae3bb90379091b77d8" Oct 30 06:26:30.817486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:30.817464 6171 scope.go:122] "RemoveContainer" containerID="5c151b30e04348c228d7c50334e386d14a63728ae89255349fb49dedcfa39462" Oct 30 06:26:34.048194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.048120 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2"] Oct 30 06:26:34.062338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.062305 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-45ce7185-dzxd2"] Oct 30 06:26:34.064950 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.064933 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk"] Oct 30 06:26:34.067337 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.067312 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc55c08a-ggvwk"] Oct 30 06:26:34.754666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.754630 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt"] Oct 30 06:26:34.755101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755088 6171 cpu_manager.go:414] "containerMap: removing container" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" Oct 30 06:26:34.755156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755102 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" Oct 30 06:26:34.755156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755121 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" Oct 30 06:26:34.755156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755126 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" Oct 30 06:26:34.755156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755138 6171 cpu_manager.go:414] "containerMap: removing container" podUID="e77cc2f1-f992-4fe7-8d03-b64444a8e307" containerName="revoke-keys" Oct 30 06:26:34.755156 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755143 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e77cc2f1-f992-4fe7-8d03-b64444a8e307" containerName="revoke-keys" Oct 30 06:26:34.755378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755214 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed69623-150a-4bc7-864b-b1bd7e822605" containerName="istio-proxy" Oct 30 06:26:34.755378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755223 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6da28380-2c55-46e6-9d73-4656683abb32" containerName="payload-pre-processing" Oct 30 06:26:34.755378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.755231 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="e77cc2f1-f992-4fe7-8d03-b64444a8e307" containerName="revoke-keys" Oct 30 06:26:34.760511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.760493 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.777824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.777798 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:34.778058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.778024 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt"] Oct 30 06:26:34.789206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.789182 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:34.789319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.789212 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:26:34.789319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.789285 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:34.830270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.830246 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.830382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.830364 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.830455 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.830440 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x92zs\" (UniqueName: \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.931542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.931507 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x92zs\" (UniqueName: \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.931716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.931573 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.931716 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.931645 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.932378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.932360 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.934108 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.934080 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:34.957904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:34.957874 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x92zs\" (UniqueName: \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\") pod \"maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:35.074020 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:35.073935 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:35.397754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:35.397727 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt"] Oct 30 06:26:35.398667 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:26:35.398641 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2261aa2_49ea_4681_af64_5a27d66d2393.slice/crio-0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9 WatchSource:0}: Error finding container 0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9: Status 404 returned error can't find the container with id 0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9 Oct 30 06:26:35.461649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:35.461621 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" event={"ID":"b2261aa2-49ea-4681-af64-5a27d66d2393","Type":"ContainerStarted","Data":"0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9"} Oct 30 06:26:35.670806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:35.670743 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ea72343-0073-4b22-9931-f3b178d56bac" path="/var/lib/kubelet/pods/2ea72343-0073-4b22-9931-f3b178d56bac/volumes" Oct 30 06:26:35.671170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:35.671156 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc673e0a-d7a4-4e70-b918-af7015ca5150" path="/var/lib/kubelet/pods/bc673e0a-d7a4-4e70-b918-af7015ca5150/volumes" Oct 30 06:26:36.041243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:36.041210 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv"] Oct 30 06:26:36.055143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:36.055116 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-14fa4301-kk5lv"] Oct 30 06:26:36.470003 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:36.469917 6171 generic.go:356] "Generic (PLEG): container finished" podID="b2261aa2-49ea-4681-af64-5a27d66d2393" containerID="717b1a87b1cbd13499f040ce0ea0cf22c6b4ba01bd8c110c2c9e2d4545ac904f" exitCode=0 Oct 30 06:26:36.470415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:36.470009 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" event={"ID":"b2261aa2-49ea-4681-af64-5a27d66d2393","Type":"ContainerDied","Data":"717b1a87b1cbd13499f040ce0ea0cf22c6b4ba01bd8c110c2c9e2d4545ac904f"} Oct 30 06:26:37.614937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.614905 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:37.671102 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.671073 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009a0372-24cb-459e-a1d0-e5ec1f5c1922" path="/var/lib/kubelet/pods/009a0372-24cb-459e-a1d0-e5ec1f5c1922/volumes" Oct 30 06:26:37.671712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.671688 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\") pod \"b2261aa2-49ea-4681-af64-5a27d66d2393\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " Oct 30 06:26:37.671830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.671730 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\" (UniqueName: \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\") pod \"b2261aa2-49ea-4681-af64-5a27d66d2393\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " Oct 30 06:26:37.671964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.671943 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\") pod \"b2261aa2-49ea-4681-af64-5a27d66d2393\" (UID: \"b2261aa2-49ea-4681-af64-5a27d66d2393\") " Oct 30 06:26:37.672363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.672326 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca" pod "b2261aa2-49ea-4681-af64-5a27d66d2393" (UID: "b2261aa2-49ea-4681-af64-5a27d66d2393"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:37.674093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.674067 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs" pod "b2261aa2-49ea-4681-af64-5a27d66d2393" (UID: "b2261aa2-49ea-4681-af64-5a27d66d2393"). InnerVolumeSpecName "kube-api-access-x92zs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:37.674192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.674151 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca" pod "b2261aa2-49ea-4681-af64-5a27d66d2393" (UID: "b2261aa2-49ea-4681-af64-5a27d66d2393"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:37.772956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.772903 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:37.772956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.772925 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x92zs\" (UniqueName: \"kubernetes.io/projected/b2261aa2-49ea-4681-af64-5a27d66d2393-kube-api-access-x92zs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:37.772956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:37.772935 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2261aa2-49ea-4681-af64-5a27d66d2393-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:38.483820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:38.483790 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" Oct 30 06:26:38.484016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:38.483823 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt" event={"ID":"b2261aa2-49ea-4681-af64-5a27d66d2393","Type":"ContainerDied","Data":"0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9"} Oct 30 06:26:38.484016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:38.483871 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6fe018ff1adf06c14c888cf42c195a01fbf0507a901437dc78e5f7ee0363f9" Oct 30 06:26:38.577904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:38.577872 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:39.536967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:39.536934 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 06:26:39.551413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:39.551383 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 06:26:45.149550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.149516 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 06:26:45.184947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.184914 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 06:26:45.487626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.487589 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2"] Oct 30 06:26:45.488122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.488101 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b2261aa2-49ea-4681-af64-5a27d66d2393" containerName="revoke-keys" Oct 30 06:26:45.488122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.488118 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2261aa2-49ea-4681-af64-5a27d66d2393" containerName="revoke-keys" Oct 30 06:26:45.488300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.488188 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2261aa2-49ea-4681-af64-5a27d66d2393" containerName="revoke-keys" Oct 30 06:26:45.492557 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.492537 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.511267 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.511243 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:26:45.511430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.511384 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:26:45.512219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.512186 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:26:45.513179 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.513158 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:26:45.515477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.515452 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2"] Oct 30 06:26:45.648414 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.648382 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.648584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.648425 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.648657 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.648621 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rp275\" (UniqueName: \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.749153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.749073 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.749153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.749114 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.749343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.749227 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rp275\" (UniqueName: \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.749903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.749878 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.751875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.751849 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.774714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.774684 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rp275\" (UniqueName: \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\") pod \"maas-api-revoke-sub-35f05329-9mcg2\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:45.807389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:45.807361 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:46.111597 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:26:46.111566 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eb8707_8a10_4e03_812a_72b490acb187.slice/crio-1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0 WatchSource:0}: Error finding container 1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0: Status 404 returned error can't find the container with id 1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0 Oct 30 06:26:46.111948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:46.111920 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2"] Oct 30 06:26:46.539091 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:46.539057 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" event={"ID":"a3eb8707-8a10-4e03-812a-72b490acb187","Type":"ContainerStarted","Data":"a682a26cd01ec4c671f3dcbc5ad3d47c34abcc5b014eeb3271f792c50c5bfe51"} Oct 30 06:26:46.539473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:46.539117 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" event={"ID":"a3eb8707-8a10-4e03-812a-72b490acb187","Type":"ContainerStarted","Data":"1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0"} Oct 30 06:26:46.585412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:46.585365 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" podStartSLOduration=1.58534621 podStartE2EDuration="1.58534621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:46.583829792 +0000 UTC m=+3265.024206985" watchObservedRunningTime="2026-10-30 06:26:46.58534621 +0000 UTC m=+3265.025723402" Oct 30 06:26:47.547423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:47.547391 6171 generic.go:356] "Generic (PLEG): container finished" podID="a3eb8707-8a10-4e03-812a-72b490acb187" containerID="a682a26cd01ec4c671f3dcbc5ad3d47c34abcc5b014eeb3271f792c50c5bfe51" exitCode=0 Oct 30 06:26:47.547850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:47.547480 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" event={"ID":"a3eb8707-8a10-4e03-812a-72b490acb187","Type":"ContainerDied","Data":"a682a26cd01ec4c671f3dcbc5ad3d47c34abcc5b014eeb3271f792c50c5bfe51"} Oct 30 06:26:48.577623 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.577586 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:26:48.704147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.704122 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:48.776707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.776683 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\") pod \"a3eb8707-8a10-4e03-812a-72b490acb187\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " Oct 30 06:26:48.776850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.776753 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\") pod \"a3eb8707-8a10-4e03-812a-72b490acb187\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " Oct 30 06:26:48.776850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.776790 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\" (UniqueName: \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\") pod \"a3eb8707-8a10-4e03-812a-72b490acb187\" (UID: \"a3eb8707-8a10-4e03-812a-72b490acb187\") " Oct 30 06:26:48.777022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.776995 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca" pod "a3eb8707-8a10-4e03-812a-72b490acb187" (UID: "a3eb8707-8a10-4e03-812a-72b490acb187"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:48.777123 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.777108 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:48.778967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.778941 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca" pod "a3eb8707-8a10-4e03-812a-72b490acb187" (UID: "a3eb8707-8a10-4e03-812a-72b490acb187"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:48.779092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.779023 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275" pod "a3eb8707-8a10-4e03-812a-72b490acb187" (UID: "a3eb8707-8a10-4e03-812a-72b490acb187"). InnerVolumeSpecName "kube-api-access-rp275". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:48.877685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.877632 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rp275\" (UniqueName: \"kubernetes.io/projected/a3eb8707-8a10-4e03-812a-72b490acb187-kube-api-access-rp275\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:48.877685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:48.877653 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3eb8707-8a10-4e03-812a-72b490acb187-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:49.561299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.561266 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" Oct 30 06:26:49.561299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.561295 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2" event={"ID":"a3eb8707-8a10-4e03-812a-72b490acb187","Type":"ContainerDied","Data":"1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0"} Oct 30 06:26:49.561515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.561337 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1cc5e5873db6a407e82bb3414d90d1cbc2a599f187e069a66a90dca64b7c56b0" Oct 30 06:26:49.917496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.917465 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:26:49.987373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.987342 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\") pod \"3de532ed-e289-4524-85ad-9e42592c7147\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " Oct 30 06:26:49.987567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.987390 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\" (UniqueName: \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\") pod \"3de532ed-e289-4524-85ad-9e42592c7147\" (UID: \"3de532ed-e289-4524-85ad-9e42592c7147\") " Oct 30 06:26:49.987780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.987756 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume" pod "3de532ed-e289-4524-85ad-9e42592c7147" (UID: "3de532ed-e289-4524-85ad-9e42592c7147"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:49.989696 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:49.989675 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh" pod "3de532ed-e289-4524-85ad-9e42592c7147" (UID: "3de532ed-e289-4524-85ad-9e42592c7147"). InnerVolumeSpecName "kube-api-access-wbnsh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:50.088661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.088600 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3de532ed-e289-4524-85ad-9e42592c7147-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.088661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.088623 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbnsh\" (UniqueName: \"kubernetes.io/projected/3de532ed-e289-4524-85ad-9e42592c7147-kube-api-access-wbnsh\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.150082 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.150003 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" podUID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" containerName="payload-processing" containerID="cri-o://e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f" gracePeriod=25 Oct 30 06:26:50.186000 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.185969 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" podUID="881a8306-55c5-47bf-a93b-dbced02b54bc" containerName="payload-pre-processing" containerID="cri-o://7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1" gracePeriod=25 Oct 30 06:26:50.438847 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.438811 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 06:26:50.453867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.453845 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 06:26:50.572192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.572159 6171 generic.go:356] "Generic (PLEG): container finished" podID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" containerID="e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f" exitCode=0 Oct 30 06:26:50.572387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.572220 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" Oct 30 06:26:50.572387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.572229 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" event={"ID":"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0","Type":"ContainerDied","Data":"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f"} Oct 30 06:26:50.572387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.572302 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6" event={"ID":"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0","Type":"ContainerDied","Data":"4da1e5bceee0d5221ba313c4970a0f532f576ad53f153211ea3e9f9f0977abe9"} Oct 30 06:26:50.572387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.572326 6171 scope.go:122] "RemoveContainer" containerID="e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f" Oct 30 06:26:50.573715 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.573688 6171 generic.go:356] "Generic (PLEG): container finished" podID="881a8306-55c5-47bf-a93b-dbced02b54bc" containerID="7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1" exitCode=0 Oct 30 06:26:50.573854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.573758 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" Oct 30 06:26:50.573854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.573772 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" event={"ID":"881a8306-55c5-47bf-a93b-dbced02b54bc","Type":"ContainerDied","Data":"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1"} Oct 30 06:26:50.573854 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.573823 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp" event={"ID":"881a8306-55c5-47bf-a93b-dbced02b54bc","Type":"ContainerDied","Data":"9877fed7340c7f02cc8e888f874a731e152253c262575ec430d852acdc20e44c"} Oct 30 06:26:50.575420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.575397 6171 generic.go:356] "Generic (PLEG): container finished" podID="3de532ed-e289-4524-85ad-9e42592c7147" containerID="bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a" exitCode=0 Oct 30 06:26:50.575546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.575426 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" event={"ID":"3de532ed-e289-4524-85ad-9e42592c7147","Type":"ContainerDied","Data":"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a"} Oct 30 06:26:50.575546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.575439 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" Oct 30 06:26:50.575546 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.575451 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z" event={"ID":"3de532ed-e289-4524-85ad-9e42592c7147","Type":"ContainerDied","Data":"64a00a25672ec28d88a1a6f5bf8c5c30dfed9726112d34406d85adb89ce91222"} Oct 30 06:26:50.583770 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.583755 6171 scope.go:122] "RemoveContainer" containerID="e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f" Oct 30 06:26:50.584087 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:50.584038 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f\": container with ID starting with e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f not found: ID does not exist" containerID="e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f" Oct 30 06:26:50.584173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.584096 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f"} err="failed to get container status \"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f\": rpc error: code = NotFound desc = could not find container \"e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f\": container with ID starting with e59e72ce30e94e1272c985eaef2873624bdbc35c595d24dceca2815dffa41c5f not found: ID does not exist" Oct 30 06:26:50.584173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.584112 6171 scope.go:122] "RemoveContainer" containerID="7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1" Oct 30 06:26:50.596199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.593459 6171 scope.go:122] "RemoveContainer" containerID="7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1" Oct 30 06:26:50.596425 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:50.596393 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1\": container with ID starting with 7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1 not found: ID does not exist" containerID="7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1" Oct 30 06:26:50.596550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596427 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1"} err="failed to get container status \"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1\": rpc error: code = NotFound desc = could not find container \"7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1\": container with ID starting with 7c52e948d565af9821e9c915f8e3a8f50cec93c1a9c7e8172bc409ab3a6d9ca1 not found: ID does not exist" Oct 30 06:26:50.596550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596449 6171 scope.go:122] "RemoveContainer" containerID="bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a" Oct 30 06:26:50.596550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596517 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\" (UniqueName: \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\") pod \"881a8306-55c5-47bf-a93b-dbced02b54bc\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " Oct 30 06:26:50.596732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596659 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\" (UniqueName: \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\") pod \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " Oct 30 06:26:50.596732 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596704 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\") pod \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\" (UID: \"fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0\") " Oct 30 06:26:50.596852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.596773 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\") pod \"881a8306-55c5-47bf-a93b-dbced02b54bc\" (UID: \"881a8306-55c5-47bf-a93b-dbced02b54bc\") " Oct 30 06:26:50.597076 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.597026 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume" pod "fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" (UID: "fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:50.597400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.597376 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume" pod "881a8306-55c5-47bf-a93b-dbced02b54bc" (UID: "881a8306-55c5-47bf-a93b-dbced02b54bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:26:50.598811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.598788 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl" pod "881a8306-55c5-47bf-a93b-dbced02b54bc" (UID: "881a8306-55c5-47bf-a93b-dbced02b54bc"). InnerVolumeSpecName "kube-api-access-snkrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:50.598911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.598819 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl" pod "fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" (UID: "fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0"). InnerVolumeSpecName "kube-api-access-9fpvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:50.607869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.607849 6171 scope.go:122] "RemoveContainer" containerID="bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a" Oct 30 06:26:50.608142 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:50.608125 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a\": container with ID starting with bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a not found: ID does not exist" containerID="bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a" Oct 30 06:26:50.608198 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.608148 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a"} err="failed to get container status \"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a\": rpc error: code = NotFound desc = could not find container \"bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a\": container with ID starting with bef5a46b2e411a6417d6d92defdb831fb7e7fe53e6a8f1b70eff7b409be2351a not found: ID does not exist" Oct 30 06:26:50.647232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.647208 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:26:50.653814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.653796 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-f6302c-5b5674d455-75n5z"] Oct 30 06:26:50.697450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.697427 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fpvl\" (UniqueName: \"kubernetes.io/projected/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-kube-api-access-9fpvl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.697450 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.697450 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.697592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.697464 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/881a8306-55c5-47bf-a93b-dbced02b54bc-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.697592 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.697477 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snkrl\" (UniqueName: \"kubernetes.io/projected/881a8306-55c5-47bf-a93b-dbced02b54bc-kube-api-access-snkrl\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:50.931242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.931209 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 06:26:50.935475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.935451 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-c78066-6cbd5bf6f5-rv9b6"] Oct 30 06:26:50.974765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.974731 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 06:26:50.979766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:50.979743 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-c78066-6b57567c95-8fccp"] Oct 30 06:26:51.670100 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:51.670070 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3de532ed-e289-4524-85ad-9e42592c7147" path="/var/lib/kubelet/pods/3de532ed-e289-4524-85ad-9e42592c7147/volumes" Oct 30 06:26:51.670427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:51.670414 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="881a8306-55c5-47bf-a93b-dbced02b54bc" path="/var/lib/kubelet/pods/881a8306-55c5-47bf-a93b-dbced02b54bc/volumes" Oct 30 06:26:51.670751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:51.670739 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" path="/var/lib/kubelet/pods/fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0/volumes" Oct 30 06:26:54.557104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.557028 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" containerID="cri-o://7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443" gracePeriod=45 Oct 30 06:26:54.570459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.570430 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" containerID="cri-o://f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9" gracePeriod=45 Oct 30 06:26:54.813992 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.813926 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:26:54.839258 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.839236 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:26:54.936957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.936922 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.937150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.936978 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937025 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.937150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937073 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.937150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937110 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\" (UniqueName: \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.937150 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937140 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937188 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\" (UniqueName: \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937215 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\" (UniqueName: \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937254 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937298 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\") pod \"7165b02e-9bec-423a-9040-55cae52fe2fb\" (UID: \"7165b02e-9bec-423a-9040-55cae52fe2fb\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937329 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937349 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.937459 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937375 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\") pod \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\" (UID: \"f6056add-7ae7-4e66-98cb-105e8c75b4e7\") " Oct 30 06:26:54.938659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937879 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:54.938659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.937931 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.938659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.938168 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.938659 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.938387 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.939268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.939232 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.939448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.939418 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.940151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.940111 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:54.940506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.940481 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.940643 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.940583 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:26:54.940819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.940789 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "kube-api-access-6mzqv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:54.940941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.940881 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d" pod "f6056add-7ae7-4e66-98cb-105e8c75b4e7" (UID: "f6056add-7ae7-4e66-98cb-105e8c75b4e7"). InnerVolumeSpecName "kube-api-access-zc66d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:26:54.941511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:54.941491 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs" pod "7165b02e-9bec-423a-9040-55cae52fe2fb" (UID: "7165b02e-9bec-423a-9040-55cae52fe2fb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:26:55.039116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039082 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039113 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039121 6171 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-dshm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039130 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc66d\" (UniqueName: \"kubernetes.io/projected/f6056add-7ae7-4e66-98cb-105e8c75b4e7-kube-api-access-zc66d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039158 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039167 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mzqv\" (UniqueName: \"kubernetes.io/projected/7165b02e-9bec-423a-9040-55cae52fe2fb-kube-api-access-6mzqv\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039175 6171 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7165b02e-9bec-423a-9040-55cae52fe2fb-tls-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039183 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039191 6171 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7165b02e-9bec-423a-9040-55cae52fe2fb-model-cache\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039198 6171 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-home\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.039353 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.039204 6171 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6056add-7ae7-4e66-98cb-105e8c75b4e7-tmp-dir\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:26:55.613864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.613826 6171 generic.go:356] "Generic (PLEG): container finished" podID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerID="7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443" exitCode=0 Oct 30 06:26:55.614294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.613871 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" Oct 30 06:26:55.614294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.613906 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" event={"ID":"7165b02e-9bec-423a-9040-55cae52fe2fb","Type":"ContainerDied","Data":"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443"} Oct 30 06:26:55.614294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.613950 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7" event={"ID":"7165b02e-9bec-423a-9040-55cae52fe2fb","Type":"ContainerDied","Data":"686484f29c3ba1c1060cb1a1403eeeca70d992ba631ec291dad18dffcca2eb1b"} Oct 30 06:26:55.614294 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.613967 6171 scope.go:122] "RemoveContainer" containerID="7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443" Oct 30 06:26:55.615610 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.615591 6171 generic.go:356] "Generic (PLEG): container finished" podID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerID="f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9" exitCode=0 Oct 30 06:26:55.615720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.615638 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" Oct 30 06:26:55.615720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.615683 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" event={"ID":"f6056add-7ae7-4e66-98cb-105e8c75b4e7","Type":"ContainerDied","Data":"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9"} Oct 30 06:26:55.615720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.615711 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n" event={"ID":"f6056add-7ae7-4e66-98cb-105e8c75b4e7","Type":"ContainerDied","Data":"ae41e410add30d99b009cafbfa05a975dada4da4cb743f6b13c931f37365d642"} Oct 30 06:26:55.624759 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.624712 6171 scope.go:122] "RemoveContainer" containerID="7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443" Oct 30 06:26:55.624999 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:55.624969 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443\": container with ID starting with 7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443 not found: ID does not exist" containerID="7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443" Oct 30 06:26:55.625134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.625011 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443"} err="failed to get container status \"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443\": rpc error: code = NotFound desc = could not find container \"7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443\": container with ID starting with 7b3053afb7c5867ddbc6a06756befebf38e2cf9cde75036b799a36e3c8cae443 not found: ID does not exist" Oct 30 06:26:55.625134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.625035 6171 scope.go:122] "RemoveContainer" containerID="f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9" Oct 30 06:26:55.634522 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.634502 6171 scope.go:122] "RemoveContainer" containerID="f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9" Oct 30 06:26:55.634758 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:55.634743 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9\": container with ID starting with f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9 not found: ID does not exist" containerID="f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9" Oct 30 06:26:55.634825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.634764 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9"} err="failed to get container status \"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9\": rpc error: code = NotFound desc = could not find container \"f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9\": container with ID starting with f86859536b921369c23da42fcf31d3ac99597cbad49124485c6be73214186fd9 not found: ID does not exist" Oct 30 06:26:55.674066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.674031 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 06:26:55.686574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.686556 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/distinct-2-w2-c78066-kserve-8449dd6d78-g72b7"] Oct 30 06:26:55.717301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.717281 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 06:26:55.721193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:55.721174 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c78066/premium-w2-c78066-kserve-7c775786fd-zl65n"] Oct 30 06:26:56.378632 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:26:56.378601 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:26:57.669843 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:57.669803 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" path="/var/lib/kubelet/pods/7165b02e-9bec-423a-9040-55cae52fe2fb/volumes" Oct 30 06:26:57.670392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:57.670355 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" path="/var/lib/kubelet/pods/f6056add-7ae7-4e66-98cb-105e8c75b4e7/volumes" Oct 30 06:26:58.526385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:26:58.526346 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:27:01.368001 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:01.367960 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt"] Oct 30 06:27:01.381891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:01.381867 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-c78066-2vqlt"] Oct 30 06:27:01.670408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:01.670329 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2261aa2-49ea-4681-af64-5a27d66d2393" path="/var/lib/kubelet/pods/b2261aa2-49ea-4681-af64-5a27d66d2393/volumes" Oct 30 06:27:07.542630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:07.542593 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:27:15.459748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.459715 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:15.460281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460262 6171 cpu_manager.go:414] "containerMap: removing container" podUID="881a8306-55c5-47bf-a93b-dbced02b54bc" containerName="payload-pre-processing" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460282 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="881a8306-55c5-47bf-a93b-dbced02b54bc" containerName="payload-pre-processing" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460296 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460301 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460318 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460323 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460339 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a3eb8707-8a10-4e03-812a-72b490acb187" containerName="revoke-keys" Oct 30 06:27:15.460341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460344 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3eb8707-8a10-4e03-812a-72b490acb187" containerName="revoke-keys" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460362 6171 cpu_manager.go:414] "containerMap: removing container" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460367 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460377 6171 cpu_manager.go:414] "containerMap: removing container" podUID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" containerName="payload-processing" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460382 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" containerName="payload-processing" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460445 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="881a8306-55c5-47bf-a93b-dbced02b54bc" containerName="payload-pre-processing" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460454 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6056add-7ae7-4e66-98cb-105e8c75b4e7" containerName="main" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460461 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3de532ed-e289-4524-85ad-9e42592c7147" containerName="payload-processing" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460471 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3eb8707-8a10-4e03-812a-72b490acb187" containerName="revoke-keys" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460476 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="7165b02e-9bec-423a-9040-55cae52fe2fb" containerName="main" Oct 30 06:27:15.460580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.460482 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdfe32e8-d06f-4f3f-8cff-d7ddd1b6fcf0" containerName="payload-processing" Oct 30 06:27:15.465338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.465322 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.494289 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.494260 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dcmpd\"" Oct 30 06:27:15.519391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.519360 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:15.521849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.521829 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:27:15.545509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.545476 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.545655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.545601 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.545733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.545686 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtx5k\" (UniqueName: \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.584033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.584010 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:27:15.584149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.584081 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:27:15.647026 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.646998 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.647146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.647094 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtx5k\" (UniqueName: \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.647202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.647157 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.649566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.649545 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.649653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.649616 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.672426 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.672401 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtx5k\" (UniqueName: \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\") pod \"maas-controller-5974d5d96-srwds\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:15.777746 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:15.777726 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:16.078302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:16.078280 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:16.079292 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:27:16.079255 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda90a72e5_0ddd_4d27_8691_8b343a908a9d.slice/crio-a46428446da882301509d65a29a9c01d52012f3b0adc72030bd4511007b65774 WatchSource:0}: Error finding container a46428446da882301509d65a29a9c01d52012f3b0adc72030bd4511007b65774: Status 404 returned error can't find the container with id a46428446da882301509d65a29a9c01d52012f3b0adc72030bd4511007b65774 Oct 30 06:27:16.764494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:16.764461 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-srwds" event={"ID":"a90a72e5-0ddd-4d27-8691-8b343a908a9d","Type":"ContainerStarted","Data":"a46428446da882301509d65a29a9c01d52012f3b0adc72030bd4511007b65774"} Oct 30 06:27:17.031009 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.030929 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45"] Oct 30 06:27:17.045327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.045293 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg"] Oct 30 06:27:17.047378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.047354 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9155c284-wwq45"] Oct 30 06:27:17.049693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.049673 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c761b88d-6v9xg"] Oct 30 06:27:17.670382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.670352 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb42d5c9-d109-478a-b05b-5a8f037ebdea" path="/var/lib/kubelet/pods/cb42d5c9-d109-478a-b05b-5a8f037ebdea/volumes" Oct 30 06:27:17.670756 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.670742 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5ffeae-7635-47ad-a73e-f168befb6d39" path="/var/lib/kubelet/pods/ce5ffeae-7635-47ad-a73e-f168befb6d39/volumes" Oct 30 06:27:17.773618 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.773583 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-srwds" event={"ID":"a90a72e5-0ddd-4d27-8691-8b343a908a9d","Type":"ContainerStarted","Data":"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653"} Oct 30 06:27:17.774005 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.773768 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:17.831121 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:17.831080 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5974d5d96-srwds" podStartSLOduration=1.230662486 podStartE2EDuration="2.831068968s" totalImagesPullingTime="1.600406482s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:17.830185368 +0000 UTC m=+3296.270562561" watchObservedRunningTime="2026-10-30 06:27:17.831068968 +0000 UTC m=+3296.271446155" Oct 30 06:27:19.031909 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:19.031873 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5"] Oct 30 06:27:19.045749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:19.045720 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6a043a72-fk9j5"] Oct 30 06:27:19.670648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:19.670616 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9651c604-5fb3-4494-b6d2-05f4794ce49e" path="/var/lib/kubelet/pods/9651c604-5fb3-4494-b6d2-05f4794ce49e/volumes" Oct 30 06:27:26.379432 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:27:26.379387 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:27:28.783055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:28.783015 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:30.890853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.890823 6171 scope.go:122] "RemoveContainer" containerID="6a482c57bf1789c731eb942497c65eb8664149dfc8e03a6db34e831e3890460c" Oct 30 06:27:30.901148 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.901128 6171 scope.go:122] "RemoveContainer" containerID="0cc02518c8004362e68588a27b589bad5213cd452675711957c34314f48a7962" Oct 30 06:27:30.910667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.910643 6171 scope.go:122] "RemoveContainer" containerID="8b8ebe65423382c5b9632e4f630db483612b493950dc58ee6b2de752558be3b9" Oct 30 06:27:30.919244 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.919222 6171 scope.go:122] "RemoveContainer" containerID="43ab56711b1a77f89b6a433ee96eaaa5716f3adf56d8cc1e42a23b36326a30c2" Oct 30 06:27:30.929830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.929813 6171 scope.go:122] "RemoveContainer" containerID="d3e1a860fc9e8778347f28d81cf8ab202c55dd44075701931664229e553db2b6" Oct 30 06:27:30.938553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.938534 6171 scope.go:122] "RemoveContainer" containerID="c0a6880f9b91b8fe84b0a4883b7b932eceefdb5d0492bdb365c2cdec9cd5dca7" Oct 30 06:27:30.946538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.946519 6171 scope.go:122] "RemoveContainer" containerID="d760166655721e0c167673b4b69ba8fcc5deb72e45ec421a2a20fbf4ec8b38e5" Oct 30 06:27:30.954834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:30.954816 6171 scope.go:122] "RemoveContainer" containerID="6e042b71ec72c048d66df38d0c5641ede560c11bede47540a5c0737389c7937a" Oct 30 06:27:36.559057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.558998 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2"] Oct 30 06:27:36.564798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.564776 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.590263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.590203 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:27:36.605980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.605910 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:27:36.605980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.605930 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:27:36.605980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.605945 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:27:36.608574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.608551 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2"] Oct 30 06:27:36.724147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.724108 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prms5\" (UniqueName: \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.724331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.724153 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.724331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.724270 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.825796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.825720 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.825796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.825775 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prms5\" (UniqueName: \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.826079 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.825882 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.826478 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.826451 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.828333 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.828314 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.851867 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.851845 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prms5\" (UniqueName: \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\") pod \"maas-api-revoke-sub-85237e45-92lt2\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:36.877367 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:36.877343 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:37.177858 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:37.177823 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2"] Oct 30 06:27:37.178675 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:27:37.178647 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb87ca31_5767_465b_8c4a_239f95c24403.slice/crio-a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9 WatchSource:0}: Error finding container a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9: Status 404 returned error can't find the container with id a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9 Oct 30 06:27:37.920110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:37.920005 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" event={"ID":"cb87ca31-5767-465b-8c4a-239f95c24403","Type":"ContainerStarted","Data":"62f957ce6da6e71013de61271ab507d04eeed161f1716c0b68859eaf85ec94fd"} Oct 30 06:27:37.920110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:37.920097 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" event={"ID":"cb87ca31-5767-465b-8c4a-239f95c24403","Type":"ContainerStarted","Data":"a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9"} Oct 30 06:27:37.966888 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:37.966840 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" podStartSLOduration=1.9668274220000002 podStartE2EDuration="1.966827422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:37.965425399 +0000 UTC m=+3316.405802591" watchObservedRunningTime="2026-10-30 06:27:37.966827422 +0000 UTC m=+3316.407204614" Oct 30 06:27:38.927714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:38.927680 6171 generic.go:356] "Generic (PLEG): container finished" podID="cb87ca31-5767-465b-8c4a-239f95c24403" containerID="62f957ce6da6e71013de61271ab507d04eeed161f1716c0b68859eaf85ec94fd" exitCode=0 Oct 30 06:27:38.928486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:38.927725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" event={"ID":"cb87ca31-5767-465b-8c4a-239f95c24403","Type":"ContainerDied","Data":"62f957ce6da6e71013de61271ab507d04eeed161f1716c0b68859eaf85ec94fd"} Oct 30 06:27:39.374390 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.374356 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h"] Oct 30 06:27:39.377632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.377611 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.408552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.408526 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h"] Oct 30 06:27:39.550059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550012 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550094 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550185 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eba797a-df5c-42d7-805d-26407499edc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550234 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7kx\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-kube-api-access-sk7kx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550276 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550316 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550308 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550342 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550398 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.550436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.550414 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eba797a-df5c-42d7-805d-26407499edc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651147 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eba797a-df5c-42d7-805d-26407499edc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651194 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7kx\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-kube-api-access-sk7kx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651219 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651244 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651263 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651294 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651313 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eba797a-df5c-42d7-805d-26407499edc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651339 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651395 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651367 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651771 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651669 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651765 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651872 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651797 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651883 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.651951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.651943 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6eba797a-df5c-42d7-805d-26407499edc7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.653740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.653719 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6eba797a-df5c-42d7-805d-26407499edc7-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.653957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.653937 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6eba797a-df5c-42d7-805d-26407499edc7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.677158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.677134 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7kx\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-kube-api-access-sk7kx\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.677272 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.677201 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6eba797a-df5c-42d7-805d-26407499edc7-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-nrm2h\" (UID: \"6eba797a-df5c-42d7-805d-26407499edc7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:39.690785 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:39.690766 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:40.015640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.015609 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h"] Oct 30 06:27:40.016590 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:27:40.016553 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eba797a_df5c_42d7_805d_26407499edc7.slice/crio-791798b8222932863a84a46cc3c5b940bed3036720dbeae3c6105be4d7bdceeb WatchSource:0}: Error finding container 791798b8222932863a84a46cc3c5b940bed3036720dbeae3c6105be4d7bdceeb: Status 404 returned error can't find the container with id 791798b8222932863a84a46cc3c5b940bed3036720dbeae3c6105be4d7bdceeb Oct 30 06:27:40.019192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.019159 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:40.019314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.019235 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:40.019314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.019281 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:27:40.062632 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.062610 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:40.157004 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.156964 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\" (UniqueName: \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\") pod \"cb87ca31-5767-465b-8c4a-239f95c24403\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " Oct 30 06:27:40.157254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.157141 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\") pod \"cb87ca31-5767-465b-8c4a-239f95c24403\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " Oct 30 06:27:40.157254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.157203 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\") pod \"cb87ca31-5767-465b-8c4a-239f95c24403\" (UID: \"cb87ca31-5767-465b-8c4a-239f95c24403\") " Oct 30 06:27:40.157492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.157470 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca" pod "cb87ca31-5767-465b-8c4a-239f95c24403" (UID: "cb87ca31-5767-465b-8c4a-239f95c24403"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:40.159225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.159199 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5" pod "cb87ca31-5767-465b-8c4a-239f95c24403" (UID: "cb87ca31-5767-465b-8c4a-239f95c24403"). InnerVolumeSpecName "kube-api-access-prms5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:40.159371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.159303 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca" pod "cb87ca31-5767-465b-8c4a-239f95c24403" (UID: "cb87ca31-5767-465b-8c4a-239f95c24403"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:40.258879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.258841 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:40.259059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.258882 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cb87ca31-5767-465b-8c4a-239f95c24403-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:40.259059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.258900 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prms5\" (UniqueName: \"kubernetes.io/projected/cb87ca31-5767-465b-8c4a-239f95c24403-kube-api-access-prms5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:40.942823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.942725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" event={"ID":"6eba797a-df5c-42d7-805d-26407499edc7","Type":"ContainerStarted","Data":"a9f51881110a362ea14a5bc0851c38acb6010ff60dcc76b61e9623845b3246e1"} Oct 30 06:27:40.942823 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.942789 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" event={"ID":"6eba797a-df5c-42d7-805d-26407499edc7","Type":"ContainerStarted","Data":"791798b8222932863a84a46cc3c5b940bed3036720dbeae3c6105be4d7bdceeb"} Oct 30 06:27:40.944163 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.944138 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" event={"ID":"cb87ca31-5767-465b-8c4a-239f95c24403","Type":"ContainerDied","Data":"a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9"} Oct 30 06:27:40.944309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.944178 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a99f66f35f52bc3905e577bd4d8a30c6e25f62daa65442b3e11559b9385915e9" Oct 30 06:27:40.944309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:40.944146 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2" Oct 30 06:27:41.012118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:41.012065 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podStartSLOduration=2.012028098 podStartE2EDuration="2.012028098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:41.009824254 +0000 UTC m=+3319.450201445" watchObservedRunningTime="2026-10-30 06:27:41.012028098 +0000 UTC m=+3319.452405287" Oct 30 06:27:41.703071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:41.691326 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:41.703599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:41.703160 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:41.703599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:41.703213 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:42.705183 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:42.705142 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:42.705688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:42.705206 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:43.703071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:43.700738 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:43.703071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:43.700806 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:44.702499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:44.702030 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:44.703080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:44.703030 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:45.702380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:45.702344 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:45.703143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:45.703112 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:46.691147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:46.691110 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" start-of-body= Oct 30 06:27:46.691357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:46.691164 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": dial tcp 10.133.1.190:15021: connect: connection refused" Oct 30 06:27:48.330423 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.330382 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:48.331028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.330997 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:48.331848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.331826 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" Oct 30 06:27:48.335809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.335789 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:48.335977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.335958 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-5974d5d96-srwds" podUID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" containerName="manager" containerID="cri-o://674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653" gracePeriod=30 Oct 30 06:27:48.612034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.612009 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:48.670071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.669425 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\") pod \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " Oct 30 06:27:48.670071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.669513 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\") pod \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " Oct 30 06:27:48.670071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.669577 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\" (UniqueName: \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\") pod \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\" (UID: \"a90a72e5-0ddd-4d27-8691-8b343a908a9d\") " Oct 30 06:27:48.673942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.673904 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert" pod "a90a72e5-0ddd-4d27-8691-8b343a908a9d" (UID: "a90a72e5-0ddd-4d27-8691-8b343a908a9d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:48.675032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.674990 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs" pod "a90a72e5-0ddd-4d27-8691-8b343a908a9d" (UID: "a90a72e5-0ddd-4d27-8691-8b343a908a9d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:48.680418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.680388 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k" pod "a90a72e5-0ddd-4d27-8691-8b343a908a9d" (UID: "a90a72e5-0ddd-4d27-8691-8b343a908a9d"). InnerVolumeSpecName "kube-api-access-rtx5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:48.771410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.771376 6171 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-webhook-cert\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:48.771410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.771405 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a90a72e5-0ddd-4d27-8691-8b343a908a9d-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:48.771410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.771415 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtx5k\" (UniqueName: \"kubernetes.io/projected/a90a72e5-0ddd-4d27-8691-8b343a908a9d-kube-api-access-rtx5k\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:27:48.997816 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.997787 6171 generic.go:356] "Generic (PLEG): container finished" podID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" containerID="674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653" exitCode=0 Oct 30 06:27:48.997968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.997858 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-srwds" Oct 30 06:27:48.997968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.997873 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-srwds" event={"ID":"a90a72e5-0ddd-4d27-8691-8b343a908a9d","Type":"ContainerDied","Data":"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653"} Oct 30 06:27:48.997968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.997930 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-srwds" event={"ID":"a90a72e5-0ddd-4d27-8691-8b343a908a9d","Type":"ContainerDied","Data":"a46428446da882301509d65a29a9c01d52012f3b0adc72030bd4511007b65774"} Oct 30 06:27:48.997968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:48.997961 6171 scope.go:122] "RemoveContainer" containerID="674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653" Oct 30 06:27:49.007928 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:49.007905 6171 scope.go:122] "RemoveContainer" containerID="674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653" Oct 30 06:27:49.008216 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:27:49.008195 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653\": container with ID starting with 674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653 not found: ID does not exist" containerID="674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653" Oct 30 06:27:49.008281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:49.008223 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653"} err="failed to get container status \"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653\": rpc error: code = NotFound desc = could not find container \"674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653\": container with ID starting with 674e25153020386ebd7080c541bcfe18d90993d33520f659d47f65aed4773653 not found: ID does not exist" Oct 30 06:27:49.056376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:49.056355 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:49.070175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:49.070152 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5974d5d96-srwds"] Oct 30 06:27:49.669856 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:49.669821 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" path="/var/lib/kubelet/pods/a90a72e5-0ddd-4d27-8691-8b343a908a9d/volumes" Oct 30 06:27:50.273262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273232 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5974d5d96-jrgzk"] Oct 30 06:27:50.273687 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273674 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" containerName="manager" Oct 30 06:27:50.273737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273690 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" containerName="manager" Oct 30 06:27:50.273737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273714 6171 cpu_manager.go:414] "containerMap: removing container" podUID="cb87ca31-5767-465b-8c4a-239f95c24403" containerName="revoke-keys" Oct 30 06:27:50.273737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273720 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb87ca31-5767-465b-8c4a-239f95c24403" containerName="revoke-keys" Oct 30 06:27:50.273850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273782 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a90a72e5-0ddd-4d27-8691-8b343a908a9d" containerName="manager" Oct 30 06:27:50.273850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.273794 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb87ca31-5767-465b-8c4a-239f95c24403" containerName="revoke-keys" Oct 30 06:27:50.282024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.282002 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.314055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.314012 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 06:27:50.314200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.314028 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 06:27:50.314200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.314028 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-dcmpd\"" Oct 30 06:27:50.314200 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.314076 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 06:27:50.316912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.316890 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5974d5d96-jrgzk"] Oct 30 06:27:50.385307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.385282 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-webhook-cert\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.385422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.385406 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-metrics-certs\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.385492 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.385460 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlck4\" (UniqueName: \"kubernetes.io/projected/eb85f3e3-0044-419f-b654-9e4d1b89626b-kube-api-access-tlck4\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.485985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.485960 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-metrics-certs\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.486132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.485997 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlck4\" (UniqueName: \"kubernetes.io/projected/eb85f3e3-0044-419f-b654-9e4d1b89626b-kube-api-access-tlck4\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.486132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.486021 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-webhook-cert\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.488533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.488511 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-metrics-certs\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.488621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.488573 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/eb85f3e3-0044-419f-b654-9e4d1b89626b-webhook-cert\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.511338 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.511313 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlck4\" (UniqueName: \"kubernetes.io/projected/eb85f3e3-0044-419f-b654-9e4d1b89626b-kube-api-access-tlck4\") pod \"maas-controller-5974d5d96-jrgzk\" (UID: \"eb85f3e3-0044-419f-b654-9e4d1b89626b\") " pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.596960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.596906 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:50.900825 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.900801 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5974d5d96-jrgzk"] Oct 30 06:27:50.902653 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:27:50.902625 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb85f3e3_0044_419f_b654_9e4d1b89626b.slice/crio-5f8be74a05a0b517715eec38b48f965d5a7829372e7caf0bab1e1a4590996853 WatchSource:0}: Error finding container 5f8be74a05a0b517715eec38b48f965d5a7829372e7caf0bab1e1a4590996853: Status 404 returned error can't find the container with id 5f8be74a05a0b517715eec38b48f965d5a7829372e7caf0bab1e1a4590996853 Oct 30 06:27:50.903953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:50.903930 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:27:51.012427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:51.012405 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-jrgzk" event={"ID":"eb85f3e3-0044-419f-b654-9e4d1b89626b","Type":"ContainerStarted","Data":"5f8be74a05a0b517715eec38b48f965d5a7829372e7caf0bab1e1a4590996853"} Oct 30 06:27:52.019904 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:52.019868 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5974d5d96-jrgzk" event={"ID":"eb85f3e3-0044-419f-b654-9e4d1b89626b","Type":"ContainerStarted","Data":"ada78b24d5fc4992cbc4b107e5bcd639a702b787afefebecd9b2ef4bede64f82"} Oct 30 06:27:52.020365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:52.019965 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:27:52.069311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:27:52.069264 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-5974d5d96-jrgzk" podStartSLOduration=1.710852864 podStartE2EDuration="2.06925316s" totalImagesPullingTime="358.400296ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:27:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:27:52.068314828 +0000 UTC m=+3330.508692020" watchObservedRunningTime="2026-10-30 06:27:52.06925316 +0000 UTC m=+3330.509630351" Oct 30 06:27:56.379564 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:27:56.379529 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:28:03.028784 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:03.028703 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5974d5d96-jrgzk" Oct 30 06:28:04.082941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:04.082909 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd"] Oct 30 06:28:04.085517 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:04.085488 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4"] Oct 30 06:28:04.087214 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:04.087188 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36f3c7aa-hzxkd"] Oct 30 06:28:04.088968 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:04.088946 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95eb23c9-q95d4"] Oct 30 06:28:05.670899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:05.670871 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dd3f2be-f258-4511-8b8c-f716b135563a" path="/var/lib/kubelet/pods/0dd3f2be-f258-4511-8b8c-f716b135563a/volumes" Oct 30 06:28:05.671308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:05.671289 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60270d63-3a5e-4599-a2b7-f02abd45edef" path="/var/lib/kubelet/pods/60270d63-3a5e-4599-a2b7-f02abd45edef/volumes" Oct 30 06:28:12.030449 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:12.030416 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45"] Oct 30 06:28:12.039605 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:12.039581 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt"] Oct 30 06:28:12.044149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:12.044128 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8c2c519e-57b45"] Oct 30 06:28:12.046021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:12.046002 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e17309f-w8gqt"] Oct 30 06:28:13.669581 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:13.669549 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d0e33cd-38e3-466d-89b0-9e667b3f4e9d" path="/var/lib/kubelet/pods/1d0e33cd-38e3-466d-89b0-9e667b3f4e9d/volumes" Oct 30 06:28:13.669964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:13.669950 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c3a8e3-e819-4cf6-b439-c7aeec9f3969" path="/var/lib/kubelet/pods/b8c3a8e3-e819-4cf6-b439-c7aeec9f3969/volumes" Oct 30 06:28:21.062233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:21.062200 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:28:26.380159 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:28:26.380122 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:28:28.008648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.008609 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv"] Oct 30 06:28:28.012602 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.012578 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.032580 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.032547 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv"] Oct 30 06:28:28.043436 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.043415 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:28.043606 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.043589 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:28.043682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.043665 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:28.043724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.043711 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:28:28.150552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.150517 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.150552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.150555 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.150774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.150588 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-222vn\" (UniqueName: \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.251591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.251547 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.251591 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.251595 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-222vn\" (UniqueName: \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.251822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.251701 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.252383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.252363 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.254110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.254092 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.277891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.277821 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-222vn\" (UniqueName: \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\") pod \"maas-api-revoke-sub-c1e271a9-kq2vv\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.325814 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.325771 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:28.628943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:28.628913 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv"] Oct 30 06:28:28.630382 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:28:28.630356 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09cd97c4_9e6e_4ee7_b177_ba2f21e32725.slice/crio-612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583 WatchSource:0}: Error finding container 612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583: Status 404 returned error can't find the container with id 612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583 Oct 30 06:28:29.281963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:29.281928 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" event={"ID":"09cd97c4-9e6e-4ee7-b177-ba2f21e32725","Type":"ContainerStarted","Data":"fcf366cbbd7bafba4d16faa83708333f37a0c9e66a518de37a8bfc96ee1ff6d5"} Oct 30 06:28:29.282379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:29.281986 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" event={"ID":"09cd97c4-9e6e-4ee7-b177-ba2f21e32725","Type":"ContainerStarted","Data":"612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583"} Oct 30 06:28:29.328634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:29.328582 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" podStartSLOduration=2.328568974 podStartE2EDuration="2.328568974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:29.326455987 +0000 UTC m=+3367.766833178" watchObservedRunningTime="2026-10-30 06:28:29.328568974 +0000 UTC m=+3367.768946166" Oct 30 06:28:30.290693 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:30.290656 6171 generic.go:356] "Generic (PLEG): container finished" podID="09cd97c4-9e6e-4ee7-b177-ba2f21e32725" containerID="fcf366cbbd7bafba4d16faa83708333f37a0c9e66a518de37a8bfc96ee1ff6d5" exitCode=0 Oct 30 06:28:30.291120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:30.290791 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" event={"ID":"09cd97c4-9e6e-4ee7-b177-ba2f21e32725","Type":"ContainerDied","Data":"fcf366cbbd7bafba4d16faa83708333f37a0c9e66a518de37a8bfc96ee1ff6d5"} Oct 30 06:28:31.028973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.028938 6171 scope.go:122] "RemoveContainer" containerID="16d12a262a0665a4adcdb049c2c5444167a7b2f60786d85d4d575c9a15a28240" Oct 30 06:28:31.038906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.038882 6171 scope.go:122] "RemoveContainer" containerID="7dbb380db7e2e9ebce1abf2685a3ba3630ffaf4a260cd714d5ccf8b5f5156a9e" Oct 30 06:28:31.048453 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.048437 6171 scope.go:122] "RemoveContainer" containerID="a33b5cef8b40cb91859147963ce62191d67da9ddff9d08d657929f5daa9efd1d" Oct 30 06:28:31.056654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.056631 6171 scope.go:122] "RemoveContainer" containerID="d998517744e841002c9e4b3afabab7ac1bb31c6ed0d355600e26314dd4a0962c" Oct 30 06:28:31.065283 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.065254 6171 scope.go:122] "RemoveContainer" containerID="30224cb68a3ff849f38104192d11e67563b0dfc566dc3fc181b9d0ee508c47df" Oct 30 06:28:31.074136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.074117 6171 scope.go:122] "RemoveContainer" containerID="92b443ba67ac0909d01349c0e913b2314cc9b6c3587802c824c12bf1c7f66c2c" Oct 30 06:28:31.082409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.082392 6171 scope.go:122] "RemoveContainer" containerID="801d3e5ad7e59356876dcd203e0b6185bbd753a083950151d5cd1c469f4dd23c" Oct 30 06:28:31.091202 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.091174 6171 scope.go:122] "RemoveContainer" containerID="c9b59038b1b72a35af1272f382ad0a409c72399e34aa25ccb20c434c059498ed" Oct 30 06:28:31.100370 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.100343 6171 scope.go:122] "RemoveContainer" containerID="738908b890c0d0e26647a4908867053bf8ed3075eec8dd9170f9152d0b7dbe67" Oct 30 06:28:31.109027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.109012 6171 scope.go:122] "RemoveContainer" containerID="dab21ff6aced8f909158366b938ef4d71ea07e6c3c3621da9fb0ed6814222ee8" Oct 30 06:28:31.117122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.117102 6171 scope.go:122] "RemoveContainer" containerID="a8a05e85b07e033987958ad5c020d92bc1f3f349d697b106fdb8b77fa8db71b5" Oct 30 06:28:31.125853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.125834 6171 scope.go:122] "RemoveContainer" containerID="43f43448f56c9f18ba4cbe8cf1ffa97cfc1b27192fd4b50bbeadd5464c578acb" Oct 30 06:28:31.437949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.437927 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:31.580958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.580874 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\") pod \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " Oct 30 06:28:31.580958 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.580914 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\") pod \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " Oct 30 06:28:31.581247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.581036 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\" (UniqueName: \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\") pod \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\" (UID: \"09cd97c4-9e6e-4ee7-b177-ba2f21e32725\") " Oct 30 06:28:31.581341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.581314 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca" pod "09cd97c4-9e6e-4ee7-b177-ba2f21e32725" (UID: "09cd97c4-9e6e-4ee7-b177-ba2f21e32725"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:31.583188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.583169 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca" pod "09cd97c4-9e6e-4ee7-b177-ba2f21e32725" (UID: "09cd97c4-9e6e-4ee7-b177-ba2f21e32725"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:28:31.583300 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.583197 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn" pod "09cd97c4-9e6e-4ee7-b177-ba2f21e32725" (UID: "09cd97c4-9e6e-4ee7-b177-ba2f21e32725"). InnerVolumeSpecName "kube-api-access-222vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:31.682322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.682297 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-222vn\" (UniqueName: \"kubernetes.io/projected/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-kube-api-access-222vn\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:31.682322 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.682321 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:31.682479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:31.682331 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09cd97c4-9e6e-4ee7-b177-ba2f21e32725-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:32.304504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:32.304471 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" Oct 30 06:28:32.304691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:32.304512 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv" event={"ID":"09cd97c4-9e6e-4ee7-b177-ba2f21e32725","Type":"ContainerDied","Data":"612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583"} Oct 30 06:28:32.304691 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:32.304587 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="612931c3ff7d9f75de94e0fcc04c407d7ae8d20c71e730712f0296cf0fb39583" Oct 30 06:28:39.153063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.153006 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:28:39.565589 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.565548 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p"] Oct 30 06:28:39.566583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.566546 6171 cpu_manager.go:414] "containerMap: removing container" podUID="09cd97c4-9e6e-4ee7-b177-ba2f21e32725" containerName="revoke-keys" Oct 30 06:28:39.566775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.566759 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09cd97c4-9e6e-4ee7-b177-ba2f21e32725" containerName="revoke-keys" Oct 30 06:28:39.567032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.567012 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="09cd97c4-9e6e-4ee7-b177-ba2f21e32725" containerName="revoke-keys" Oct 30 06:28:39.571265 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.571244 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.593569 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.590916 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p"] Oct 30 06:28:39.759525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759492 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.759525 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759527 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hc7b\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-kube-api-access-4hc7b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.759802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759615 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a3a294-bca7-4761-8947-f0b5d4125a72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.759802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759641 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.759802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759659 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.759802 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759743 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.760093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759851 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.760093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759874 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.760093 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.759903 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.860952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.860871 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.860952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.860907 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.860952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.860926 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.860952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.860952 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861008 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861036 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hc7b\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-kube-api-access-4hc7b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861131 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a3a294-bca7-4761-8947-f0b5d4125a72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861169 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861196 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861409 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861364 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-credential-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861800 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861749 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4a3a294-bca7-4761-8947-f0b5d4125a72-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.861985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.861961 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-certs\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.862096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.862024 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-data\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.862096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.862069 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-workload-socket\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.863552 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.863529 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-podinfo\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.863969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.863949 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-envoy\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.888290 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.888260 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-istio-token\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:39.888419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:39.888317 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hc7b\" (UniqueName: \"kubernetes.io/projected/f4a3a294-bca7-4761-8947-f0b5d4125a72-kube-api-access-4hc7b\") pod \"maas-default-gateway-openshift-default-58847f9fc5-mbq7p\" (UID: \"f4a3a294-bca7-4761-8947-f0b5d4125a72\") " pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:40.186795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:40.186709 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:40.513616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:40.513584 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p"] Oct 30 06:28:40.515912 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:28:40.515877 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a3a294_bca7_4761_8947_f0b5d4125a72.slice/crio-254a3f678cb7b040ee43b613f21553e1a0de18c30efbb5c82bc17b8f00ab4803 WatchSource:0}: Error finding container 254a3f678cb7b040ee43b613f21553e1a0de18c30efbb5c82bc17b8f00ab4803: Status 404 returned error can't find the container with id 254a3f678cb7b040ee43b613f21553e1a0de18c30efbb5c82bc17b8f00ab4803 Oct 30 06:28:40.518416 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:40.518374 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:40.518540 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:40.518465 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:40.518540 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:40.518512 6171 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 06:28:41.362718 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:41.362680 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" event={"ID":"f4a3a294-bca7-4761-8947-f0b5d4125a72","Type":"ContainerStarted","Data":"2abbcbcab7ffbc848c42744913e680f85ef4e870de8159d93d2df195de8adef7"} Oct 30 06:28:41.363195 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:41.362739 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" event={"ID":"f4a3a294-bca7-4761-8947-f0b5d4125a72","Type":"ContainerStarted","Data":"254a3f678cb7b040ee43b613f21553e1a0de18c30efbb5c82bc17b8f00ab4803"} Oct 30 06:28:41.436118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:41.436019 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podStartSLOduration=2.435999753 podStartE2EDuration="2.435999753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:41.434482759 +0000 UTC m=+3379.874859955" watchObservedRunningTime="2026-10-30 06:28:41.435999753 +0000 UTC m=+3379.876376947" Oct 30 06:28:42.201071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:42.197384 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:42.206319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:42.206288 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:42.206846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:42.206345 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:43.200070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:43.197501 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:43.200070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:43.197573 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:43.857905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:43.857870 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:28:44.191582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:44.191483 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:44.191879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:44.191846 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:45.190499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:45.190460 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:45.191130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:45.191096 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:46.187498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:46.187456 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:46.188159 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:46.188124 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:47.041429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.041389 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq"] Oct 30 06:28:47.059998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.059924 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5"] Oct 30 06:28:47.064155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.064094 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb1505f7-hmxdq"] Oct 30 06:28:47.067835 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.067802 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-264cef08-7b7h5"] Oct 30 06:28:47.187319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.187289 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:47.187564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.187528 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:47.671209 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.671177 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="487017ed-7702-4a96-abec-94fc1174a84d" path="/var/lib/kubelet/pods/487017ed-7702-4a96-abec-94fc1174a84d/volumes" Oct 30 06:28:47.671644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:47.671629 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ceaab93e-f29f-494c-9d8f-2a70112d4a06" path="/var/lib/kubelet/pods/ceaab93e-f29f-494c-9d8f-2a70112d4a06/volumes" Oct 30 06:28:48.058940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.058902 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6"] Oct 30 06:28:48.063401 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.063376 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.083655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.083629 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6"] Oct 30 06:28:48.094684 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.094657 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:48.094808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.094696 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:48.096749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.096728 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:48.096861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.096756 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:28:48.187626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.187594 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" start-of-body= Oct 30 06:28:48.187764 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.187644 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": dial tcp 10.133.1.193:15021: connect: connection refused" Oct 30 06:28:48.223951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.223928 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.224090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.223973 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.224146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.224112 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5gmp\" (UniqueName: \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.325274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.325205 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5gmp\" (UniqueName: \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.325274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.325257 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.325429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.325295 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.325989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.325958 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.327707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.327682 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.353780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.353755 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5gmp\" (UniqueName: \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\") pod \"maas-api-revoke-sub-2e1f5b5f-q5tb6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.378373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.378349 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:48.714656 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:28:48.714630 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03c1432_c498_4254_a936_e59d6a85d8d6.slice/crio-f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9 WatchSource:0}: Error finding container f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9: Status 404 returned error can't find the container with id f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9 Oct 30 06:28:48.715418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:48.715389 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6"] Oct 30 06:28:49.452465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:49.452364 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" event={"ID":"d03c1432-c498-4254-a936-e59d6a85d8d6","Type":"ContainerStarted","Data":"fdaf24209445ab7b7ea8241f6f0218cee093772bab1de6f35fda57c726d7ba6d"} Oct 30 06:28:49.452465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:49.452442 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" event={"ID":"d03c1432-c498-4254-a936-e59d6a85d8d6","Type":"ContainerStarted","Data":"f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9"} Oct 30 06:28:49.498240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:49.498180 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" podStartSLOduration=1.498166833 podStartE2EDuration="1.498166833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:49.497105235 +0000 UTC m=+3387.937482427" watchObservedRunningTime="2026-10-30 06:28:49.498166833 +0000 UTC m=+3387.938544025" Oct 30 06:28:50.188164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:50.188117 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:28:50.188408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:50.188182 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:28:50.462933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:50.462834 6171 generic.go:356] "Generic (PLEG): container finished" podID="d03c1432-c498-4254-a936-e59d6a85d8d6" containerID="fdaf24209445ab7b7ea8241f6f0218cee093772bab1de6f35fda57c726d7ba6d" exitCode=0 Oct 30 06:28:50.462933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:50.462912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" event={"ID":"d03c1432-c498-4254-a936-e59d6a85d8d6","Type":"ContainerDied","Data":"fdaf24209445ab7b7ea8241f6f0218cee093772bab1de6f35fda57c726d7ba6d"} Oct 30 06:28:51.230243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.230196 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-mbq7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:28:51.230497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.230267 6171 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" podUID="f4a3a294-bca7-4761-8947-f0b5d4125a72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:28:51.614199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.614162 6171 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:51.615780 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.615755 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:51.615915 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.615845 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-mbq7p" Oct 30 06:28:51.626673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.626647 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:51.658622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.658597 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\") pod \"d03c1432-c498-4254-a936-e59d6a85d8d6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " Oct 30 06:28:51.658749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.658711 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\" (UniqueName: \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\") pod \"d03c1432-c498-4254-a936-e59d6a85d8d6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " Oct 30 06:28:51.658749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.658733 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\") pod \"d03c1432-c498-4254-a936-e59d6a85d8d6\" (UID: \"d03c1432-c498-4254-a936-e59d6a85d8d6\") " Oct 30 06:28:51.659223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.659195 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca" pod "d03c1432-c498-4254-a936-e59d6a85d8d6" (UID: "d03c1432-c498-4254-a936-e59d6a85d8d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:51.660960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.660941 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca" pod "d03c1432-c498-4254-a936-e59d6a85d8d6" (UID: "d03c1432-c498-4254-a936-e59d6a85d8d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:28:51.661036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.661002 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp" pod "d03c1432-c498-4254-a936-e59d6a85d8d6" (UID: "d03c1432-c498-4254-a936-e59d6a85d8d6"). InnerVolumeSpecName "kube-api-access-z5gmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:51.760386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.760356 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:51.760386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.760385 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5gmp\" (UniqueName: \"kubernetes.io/projected/d03c1432-c498-4254-a936-e59d6a85d8d6-kube-api-access-z5gmp\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:51.760386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:51.760395 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d03c1432-c498-4254-a936-e59d6a85d8d6-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:28:52.475846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:52.475818 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" Oct 30 06:28:52.476034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:52.475850 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6" event={"ID":"d03c1432-c498-4254-a936-e59d6a85d8d6","Type":"ContainerDied","Data":"f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9"} Oct 30 06:28:52.476034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:52.475898 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7606ac814d50a8d284b44f4017e54de91d8f9e5482ad08dc63d96774ff4f7b9" Oct 30 06:28:56.379943 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:28:56.379904 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:28:59.150701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.150663 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8"] Oct 30 06:28:59.151139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.151117 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d03c1432-c498-4254-a936-e59d6a85d8d6" containerName="revoke-keys" Oct 30 06:28:59.151139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.151129 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d03c1432-c498-4254-a936-e59d6a85d8d6" containerName="revoke-keys" Oct 30 06:28:59.151245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.151197 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d03c1432-c498-4254-a936-e59d6a85d8d6" containerName="revoke-keys" Oct 30 06:28:59.154319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.154299 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.173548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.173337 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8"] Oct 30 06:28:59.184738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.184708 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:59.184738 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.184732 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:59.184919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.184736 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:59.185033 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.185019 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:28:59.225583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.225547 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbqcz\" (UniqueName: \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.225761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.225607 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.225761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.225740 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.253960 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.253928 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:28:59.326917 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.326887 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbqcz\" (UniqueName: \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.327153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.326934 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.327153 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.326979 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.327740 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.327719 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.329579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.329557 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.351871 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.351838 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbqcz\" (UniqueName: \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\") pod \"maas-api-revoke-sub-ec843018-h7nh8\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.469949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.469883 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:28:59.781796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:28:59.781759 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8"] Oct 30 06:28:59.787334 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:28:59.787298 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1952b7b0_aeb3_45a5_9f95_a2dc7c27b1bb.slice/crio-791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b WatchSource:0}: Error finding container 791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b: Status 404 returned error can't find the container with id 791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b Oct 30 06:29:00.534538 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:00.534508 6171 generic.go:356] "Generic (PLEG): container finished" podID="1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" containerID="5ccfed1ec3dd268a5ac457f7c8fd00e8fc6e50c469fa4930946de46f63f782df" exitCode=0 Oct 30 06:29:00.534931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:00.534634 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" event={"ID":"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb","Type":"ContainerDied","Data":"5ccfed1ec3dd268a5ac457f7c8fd00e8fc6e50c469fa4930946de46f63f782df"} Oct 30 06:29:00.534931 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:00.534725 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" event={"ID":"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb","Type":"ContainerStarted","Data":"791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b"} Oct 30 06:29:01.693269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.693241 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:29:01.752980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.752940 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\") pod \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " Oct 30 06:29:01.753208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.752991 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\" (UniqueName: \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\") pod \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " Oct 30 06:29:01.753208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.753118 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\") pod \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\" (UID: \"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb\") " Oct 30 06:29:01.753483 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.753455 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca" pod "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" (UID: "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:01.753620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.753576 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:01.755934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.755905 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca" pod "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" (UID: "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:01.756274 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.756242 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz" pod "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" (UID: "1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb"). InnerVolumeSpecName "kube-api-access-wbqcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:01.854808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.854725 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:01.854808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:01.854753 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbqcz\" (UniqueName: \"kubernetes.io/projected/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb-kube-api-access-wbqcz\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:02.554967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:02.554932 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" Oct 30 06:29:02.555225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:02.554932 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8" event={"ID":"1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb","Type":"ContainerDied","Data":"791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b"} Oct 30 06:29:02.555225 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:02.555096 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="791dbde38fbb3160566032d076140816f5420b450e1d65355c3471f6304ceb7b" Oct 30 06:29:04.376293 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:04.376249 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:29:04.376898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:04.376862 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:29:14.659727 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:14.659687 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:29:26.379741 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:29:26.379701 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:29:27.081208 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.081169 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr"] Oct 30 06:29:27.081587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.081575 6171 cpu_manager.go:414] "containerMap: removing container" podUID="1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" containerName="revoke-keys" Oct 30 06:29:27.081649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.081589 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" containerName="revoke-keys" Oct 30 06:29:27.081698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.081681 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" containerName="revoke-keys" Oct 30 06:29:27.084894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.084875 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.102139 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.102106 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:29:27.116252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.116220 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:29:27.116451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.116306 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:27.116451 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.116336 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:27.119461 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.119435 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr"] Oct 30 06:29:27.123918 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.123890 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.124035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.123960 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b4jd\" (UniqueName: \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.124140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.124115 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.225071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.225017 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.225270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.225103 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b4jd\" (UniqueName: \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.225270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.225133 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.225754 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.225735 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.227466 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.227447 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.250853 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.250828 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b4jd\" (UniqueName: \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\") pod \"maas-api-revoke-sub-046e1343-5xrlr\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.404012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.403937 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:27.705482 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.705450 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr"] Oct 30 06:29:27.706072 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:29:27.706022 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09577c80_9236_4833_b849_0fd9c04d833f.slice/crio-6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a WatchSource:0}: Error finding container 6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a: Status 404 returned error can't find the container with id 6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a Oct 30 06:29:27.794056 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:27.794019 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" event={"ID":"09577c80-9236-4833-b849-0fd9c04d833f","Type":"ContainerStarted","Data":"6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a"} Oct 30 06:29:28.801901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:28.801860 6171 generic.go:356] "Generic (PLEG): container finished" podID="09577c80-9236-4833-b849-0fd9c04d833f" containerID="4b4e6b8e1b083238ee6072964f0b40c1906402798a9e434a0ff00f11d442813b" exitCode=0 Oct 30 06:29:28.802350 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:28.801948 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" event={"ID":"09577c80-9236-4833-b849-0fd9c04d833f","Type":"ContainerDied","Data":"4b4e6b8e1b083238ee6072964f0b40c1906402798a9e434a0ff00f11d442813b"} Oct 30 06:29:29.948021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:29.947997 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:30.056071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.056007 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\" (UniqueName: \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\") pod \"09577c80-9236-4833-b849-0fd9c04d833f\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " Oct 30 06:29:30.056257 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.056189 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\") pod \"09577c80-9236-4833-b849-0fd9c04d833f\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " Oct 30 06:29:30.056311 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.056257 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\") pod \"09577c80-9236-4833-b849-0fd9c04d833f\" (UID: \"09577c80-9236-4833-b849-0fd9c04d833f\") " Oct 30 06:29:30.056622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.056589 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca" pod "09577c80-9236-4833-b849-0fd9c04d833f" (UID: "09577c80-9236-4833-b849-0fd9c04d833f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:30.058331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.058310 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd" pod "09577c80-9236-4833-b849-0fd9c04d833f" (UID: "09577c80-9236-4833-b849-0fd9c04d833f"). InnerVolumeSpecName "kube-api-access-2b4jd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:30.058415 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.058365 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca" pod "09577c80-9236-4833-b849-0fd9c04d833f" (UID: "09577c80-9236-4833-b849-0fd9c04d833f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:30.157808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.157783 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:30.157808 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.157806 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09577c80-9236-4833-b849-0fd9c04d833f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:30.157953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.157815 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b4jd\" (UniqueName: \"kubernetes.io/projected/09577c80-9236-4833-b849-0fd9c04d833f-kube-api-access-2b4jd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:30.816171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.816132 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" event={"ID":"09577c80-9236-4833-b849-0fd9c04d833f","Type":"ContainerDied","Data":"6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a"} Oct 30 06:29:30.816340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.816182 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6266ab24538fab2c0a16c847b0f4d13ab6b15f7ba8e5ae446fcbc0feab6e554a" Oct 30 06:29:30.816340 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:30.816140 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr" Oct 30 06:29:31.203565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.203479 6171 scope.go:122] "RemoveContainer" containerID="7338841c8133b4556d1c37daa6652e8814d1a604800d286aec1a0a3c8ed025a0" Oct 30 06:29:31.214090 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.214066 6171 scope.go:122] "RemoveContainer" containerID="273a8743fee85bc3a134c27b3699e4e85b95afb85f583f5204caec7a634c19fd" Oct 30 06:29:31.222956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.222937 6171 scope.go:122] "RemoveContainer" containerID="de3778d52d39f4f1e167d0dbf38563449adacff9ce85591872cf8007863b0ac0" Oct 30 06:29:31.230985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.230969 6171 scope.go:122] "RemoveContainer" containerID="5194d1d778c32944ca34730cfcfcdf51e6aa985c2aa0b57f3814ec5265119aae" Oct 30 06:29:31.239034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.239014 6171 scope.go:122] "RemoveContainer" containerID="f49ef91d11a3f11fc5dbe73655d7d65254fd4cf12e2259b11a38cc254ef3977b" Oct 30 06:29:31.247092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.247071 6171 scope.go:122] "RemoveContainer" containerID="9765c29a3451bc19f623830d329edc6a8f9d6df7747d588cf57b739bb89c3439" Oct 30 06:29:31.254884 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.254864 6171 scope.go:122] "RemoveContainer" containerID="2912d52867d14e65229c994911d100165fa492a0d8f195f0c32ad522870d263b" Oct 30 06:29:31.262661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:31.262643 6171 scope.go:122] "RemoveContainer" containerID="cb951db38b26a7dcdbd6af51b554ed0821e5aa1e98f5fce8dca9069e60be8f6f" Oct 30 06:29:37.481376 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.481344 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79"] Oct 30 06:29:37.481791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.481778 6171 cpu_manager.go:414] "containerMap: removing container" podUID="09577c80-9236-4833-b849-0fd9c04d833f" containerName="revoke-keys" Oct 30 06:29:37.481839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.481792 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09577c80-9236-4833-b849-0fd9c04d833f" containerName="revoke-keys" Oct 30 06:29:37.481886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.481861 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="09577c80-9236-4833-b849-0fd9c04d833f" containerName="revoke-keys" Oct 30 06:29:37.487072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.487032 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.505319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.505290 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79"] Oct 30 06:29:37.516201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.516173 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:29:37.516381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.516181 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:37.516381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.516225 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:37.516381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.516225 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:29:37.630281 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.630251 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48brs\" (UniqueName: \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.630429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.630302 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.630486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.630422 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.731371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.731338 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48brs\" (UniqueName: \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.731585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.731392 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.731585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.731439 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.732112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.732089 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.733921 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.733899 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.756850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.756816 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48brs\" (UniqueName: \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\") pod \"maas-api-revoke-sub-d3fba056-8wm79\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:37.801410 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:37.801386 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:38.103098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:38.103074 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79"] Oct 30 06:29:38.104466 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:29:38.104435 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc70a70e_b406_4277_b22d_7181efc69d21.slice/crio-18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204 WatchSource:0}: Error finding container 18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204: Status 404 returned error can't find the container with id 18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204 Oct 30 06:29:38.323663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:38.323630 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:29:38.870751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:38.870707 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" event={"ID":"dc70a70e-b406-4277-b22d-7181efc69d21","Type":"ContainerStarted","Data":"b7788dfe3e11db22a584d1dbc135c88b697526ae2682098c54cf17a8c387c1a4"} Oct 30 06:29:38.871324 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:38.870783 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" event={"ID":"dc70a70e-b406-4277-b22d-7181efc69d21","Type":"ContainerStarted","Data":"18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204"} Oct 30 06:29:38.922008 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:38.921929 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" podStartSLOduration=1.921909201 podStartE2EDuration="1.921909201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:38.917237353 +0000 UTC m=+3437.357614548" watchObservedRunningTime="2026-10-30 06:29:38.921909201 +0000 UTC m=+3437.362286395" Oct 30 06:29:39.884072 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:39.881898 6171 generic.go:356] "Generic (PLEG): container finished" podID="dc70a70e-b406-4277-b22d-7181efc69d21" containerID="b7788dfe3e11db22a584d1dbc135c88b697526ae2682098c54cf17a8c387c1a4" exitCode=0 Oct 30 06:29:39.888066 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:39.886110 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" event={"ID":"dc70a70e-b406-4277-b22d-7181efc69d21","Type":"ContainerDied","Data":"b7788dfe3e11db22a584d1dbc135c88b697526ae2682098c54cf17a8c387c1a4"} Oct 30 06:29:41.543103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.534327 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:41.623270 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.623233 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\") pod \"dc70a70e-b406-4277-b22d-7181efc69d21\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " Oct 30 06:29:41.623493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.623285 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\" (UniqueName: \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\") pod \"dc70a70e-b406-4277-b22d-7181efc69d21\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " Oct 30 06:29:41.623493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.623425 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\") pod \"dc70a70e-b406-4277-b22d-7181efc69d21\" (UID: \"dc70a70e-b406-4277-b22d-7181efc69d21\") " Oct 30 06:29:41.625430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.624443 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca" pod "dc70a70e-b406-4277-b22d-7181efc69d21" (UID: "dc70a70e-b406-4277-b22d-7181efc69d21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:41.627577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.627548 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs" pod "dc70a70e-b406-4277-b22d-7181efc69d21" (UID: "dc70a70e-b406-4277-b22d-7181efc69d21"). InnerVolumeSpecName "kube-api-access-48brs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:41.631243 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.631221 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca" pod "dc70a70e-b406-4277-b22d-7181efc69d21" (UID: "dc70a70e-b406-4277-b22d-7181efc69d21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:41.724712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.724677 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:41.724712 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.724715 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48brs\" (UniqueName: \"kubernetes.io/projected/dc70a70e-b406-4277-b22d-7181efc69d21-kube-api-access-48brs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:41.725002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.724730 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc70a70e-b406-4277-b22d-7181efc69d21-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:29:41.913403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.913314 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" event={"ID":"dc70a70e-b406-4277-b22d-7181efc69d21","Type":"ContainerDied","Data":"18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204"} Oct 30 06:29:41.913403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.913369 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18a4237d4207090eb7f389a456114917524393a647544e9e5bdab58f7ae11204" Oct 30 06:29:41.913403 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:41.913322 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79" Oct 30 06:29:48.742348 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:48.742315 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:29:53.144199 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:29:53.144164 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:29:56.379609 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:29:56.379575 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:30:00.118535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.118496 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx"] Oct 30 06:30:00.119206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.119187 6171 cpu_manager.go:414] "containerMap: removing container" podUID="dc70a70e-b406-4277-b22d-7181efc69d21" containerName="revoke-keys" Oct 30 06:30:00.119206 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.119208 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc70a70e-b406-4277-b22d-7181efc69d21" containerName="revoke-keys" Oct 30 06:30:00.119318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.119312 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc70a70e-b406-4277-b22d-7181efc69d21" containerName="revoke-keys" Oct 30 06:30:00.122499 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.122474 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.140501 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.140461 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx"] Oct 30 06:30:00.151660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.151628 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:30:00.151813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.151706 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:00.151876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.151827 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:30:00.151949 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.151879 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:00.250619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.250587 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.250804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.250626 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6ml\" (UniqueName: \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.250804 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.250726 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.352024 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.351987 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.352246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.352055 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6ml\" (UniqueName: \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.352246 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.352119 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.352869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.352770 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.354620 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.354594 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.377927 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.377862 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6ml\" (UniqueName: \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\") pod \"maas-api-revoke-sub-ae594979-kwqmx\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.443235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.443201 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:00.745664 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:00.745634 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx"] Oct 30 06:30:00.746437 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:30:00.746410 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1789d1_2b91_4faf_93cb_ce615879d701.slice/crio-ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b WatchSource:0}: Error finding container ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b: Status 404 returned error can't find the container with id ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b Oct 30 06:30:01.049083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:01.048979 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" event={"ID":"3d1789d1-2b91-4faf-93cb-ce615879d701","Type":"ContainerStarted","Data":"68e3682693a4e4dd6aa15dd48d105a5d8cc5eb6b5a766f740fcc6bbc60ac06d3"} Oct 30 06:30:01.049083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:01.049038 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" event={"ID":"3d1789d1-2b91-4faf-93cb-ce615879d701","Type":"ContainerStarted","Data":"ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b"} Oct 30 06:30:01.095752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:01.095694 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" podStartSLOduration=1.095680972 podStartE2EDuration="1.095680972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:01.09342582 +0000 UTC m=+3459.533803012" watchObservedRunningTime="2026-10-30 06:30:01.095680972 +0000 UTC m=+3459.536058164" Oct 30 06:30:02.056859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:02.056823 6171 generic.go:356] "Generic (PLEG): container finished" podID="3d1789d1-2b91-4faf-93cb-ce615879d701" containerID="68e3682693a4e4dd6aa15dd48d105a5d8cc5eb6b5a766f740fcc6bbc60ac06d3" exitCode=0 Oct 30 06:30:02.057275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:02.056912 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" event={"ID":"3d1789d1-2b91-4faf-93cb-ce615879d701","Type":"ContainerDied","Data":"68e3682693a4e4dd6aa15dd48d105a5d8cc5eb6b5a766f740fcc6bbc60ac06d3"} Oct 30 06:30:03.203562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.203537 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:03.279259 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.279229 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\" (UniqueName: \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\") pod \"3d1789d1-2b91-4faf-93cb-ce615879d701\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " Oct 30 06:30:03.279690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.279296 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\") pod \"3d1789d1-2b91-4faf-93cb-ce615879d701\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " Oct 30 06:30:03.279690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.279398 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\") pod \"3d1789d1-2b91-4faf-93cb-ce615879d701\" (UID: \"3d1789d1-2b91-4faf-93cb-ce615879d701\") " Oct 30 06:30:03.279812 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.279775 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca" pod "3d1789d1-2b91-4faf-93cb-ce615879d701" (UID: "3d1789d1-2b91-4faf-93cb-ce615879d701"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:03.282174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.282146 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca" pod "3d1789d1-2b91-4faf-93cb-ce615879d701" (UID: "3d1789d1-2b91-4faf-93cb-ce615879d701"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:03.282174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.282156 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml" pod "3d1789d1-2b91-4faf-93cb-ce615879d701" (UID: "3d1789d1-2b91-4faf-93cb-ce615879d701"). InnerVolumeSpecName "kube-api-access-qz6ml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:03.380577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.380489 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6ml\" (UniqueName: \"kubernetes.io/projected/3d1789d1-2b91-4faf-93cb-ce615879d701-kube-api-access-qz6ml\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.380577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.380527 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:03.380577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:03.380538 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3d1789d1-2b91-4faf-93cb-ce615879d701-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:04.070448 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:04.070418 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" Oct 30 06:30:04.070644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:04.070448 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx" event={"ID":"3d1789d1-2b91-4faf-93cb-ce615879d701","Type":"ContainerDied","Data":"ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b"} Oct 30 06:30:04.070644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:04.070508 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ddbef8f61940285aa0b41b53743b29bdb4d802ef028bc0500f01093ba73d4a2b" Oct 30 06:30:11.261493 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:11.261458 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:30:16.262967 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:16.262926 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:30:21.580786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.580750 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4"] Oct 30 06:30:21.581308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.581209 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3d1789d1-2b91-4faf-93cb-ce615879d701" containerName="revoke-keys" Oct 30 06:30:21.581308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.581220 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1789d1-2b91-4faf-93cb-ce615879d701" containerName="revoke-keys" Oct 30 06:30:21.581308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.581297 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1789d1-2b91-4faf-93cb-ce615879d701" containerName="revoke-keys" Oct 30 06:30:21.585714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.585695 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.605034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.605007 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4"] Oct 30 06:30:21.615573 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.615545 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:30:21.616744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.616715 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:30:21.616744 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.616735 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:21.616913 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.616827 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:21.644233 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.644200 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmrc6\" (UniqueName: \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.644391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.644237 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.644391 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.644380 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.745329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.745295 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmrc6\" (UniqueName: \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.745329 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.745331 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.745622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.745411 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.746201 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.746177 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.747993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.747968 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.784377 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.784342 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:21.791857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.791837 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmrc6\" (UniqueName: \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\") pod \"maas-api-revoke-sub-a4aef7d7-ljcw4\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:21.914890 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.914826 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:30:21.919898 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:21.919880 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:22.224550 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:22.224514 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4"] Oct 30 06:30:22.225126 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:30:22.225099 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5ffcd6d_3fb2_4b77_966a_aa5e1a96914f.slice/crio-3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef WatchSource:0}: Error finding container 3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef: Status 404 returned error can't find the container with id 3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef Oct 30 06:30:23.222861 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:23.222828 6171 generic.go:356] "Generic (PLEG): container finished" podID="f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" containerID="26198c05bf09eb89e94d89c991a5e7ad6950d19a6dee04035e346e837a2cd766" exitCode=0 Oct 30 06:30:23.223326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:23.222916 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" event={"ID":"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f","Type":"ContainerDied","Data":"26198c05bf09eb89e94d89c991a5e7ad6950d19a6dee04035e346e837a2cd766"} Oct 30 06:30:23.223326 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:23.222970 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" event={"ID":"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f","Type":"ContainerStarted","Data":"3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef"} Oct 30 06:30:24.371308 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.371284 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:24.470218 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.470189 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\") pod \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " Oct 30 06:30:24.470375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.470288 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\" (UniqueName: \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\") pod \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " Oct 30 06:30:24.470375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.470356 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\") pod \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\" (UID: \"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f\") " Oct 30 06:30:24.470737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.470707 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca" pod "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" (UID: "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:24.472437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.472415 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca" pod "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" (UID: "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:24.472437 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.472428 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6" pod "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" (UID: "f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f"). InnerVolumeSpecName "kube-api-access-wmrc6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:24.571941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.571860 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:24.571941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.571891 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:24.571941 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:24.571901 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmrc6\" (UniqueName: \"kubernetes.io/projected/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f-kube-api-access-wmrc6\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:25.236570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:25.236541 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" Oct 30 06:30:25.236768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:25.236578 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4" event={"ID":"f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f","Type":"ContainerDied","Data":"3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef"} Oct 30 06:30:25.236768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:25.236627 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d450af18ba0ec24935e8d5b766336921f10ab2f21cec48b4b7c5bc9c06c56ef" Oct 30 06:30:26.380915 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:30:26.380881 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:30:31.314786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:31.314752 6171 scope.go:122] "RemoveContainer" containerID="ae464467209683a151015be98902db215278fdc1b689a56381cfe248a89c4db7" Oct 30 06:30:31.324114 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:31.324090 6171 scope.go:122] "RemoveContainer" containerID="a14d698d08124febf29b32f4c4f1a2d223b25d19b5b412301334603d5f8a2b53" Oct 30 06:30:31.332682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:31.332661 6171 scope.go:122] "RemoveContainer" containerID="0ab805a9400a07b1bc26e734c8d37d4c2ece65315d55f9916a8fd575c0bd6de7" Oct 30 06:30:32.735852 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:32.735815 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:30:37.050511 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:37.050474 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:30:38.031734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:38.031687 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm"] Oct 30 06:30:38.045634 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:38.045591 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2878804e-dgzbm"] Oct 30 06:30:38.049349 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:38.049318 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht"] Oct 30 06:30:38.053129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:38.053098 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3c520a6f-2lrht"] Oct 30 06:30:39.721070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:39.720715 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4eec602c-90c1-48e0-9c27-8e91f7876591" path="/var/lib/kubelet/pods/4eec602c-90c1-48e0-9c27-8e91f7876591/volumes" Oct 30 06:30:39.721617 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:39.721341 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649d06a2-6652-47f9-8443-8f83193c55e0" path="/var/lib/kubelet/pods/649d06a2-6652-47f9-8443-8f83193c55e0/volumes" Oct 30 06:30:43.561167 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.561134 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2"] Oct 30 06:30:43.561686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.561670 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" containerName="revoke-keys" Oct 30 06:30:43.561751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.561688 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" containerName="revoke-keys" Oct 30 06:30:43.561799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.561773 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" containerName="revoke-keys" Oct 30 06:30:43.566469 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.566449 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.584944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.584920 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2"] Oct 30 06:30:43.596105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.596071 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:30:43.596105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.596098 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:30:43.596933 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.596916 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:30:43.597012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.596981 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:30:43.627786 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.627764 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p749h\" (UniqueName: \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.627897 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.627796 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.627956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.627939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.729105 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.729079 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.729220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.729125 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p749h\" (UniqueName: \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.729220 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.729147 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.729899 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.729815 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.731496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.731479 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.754640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.754618 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p749h\" (UniqueName: \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\") pod \"maas-api-revoke-sub-79805ca9-j8wv2\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:43.880252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:43.880191 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:44.182136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:44.182098 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2"] Oct 30 06:30:44.182938 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:30:44.182914 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d64e6e9_0ba1_47fa_afc0_d5424dfffc9e.slice/crio-c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41 WatchSource:0}: Error finding container c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41: Status 404 returned error can't find the container with id c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41 Oct 30 06:30:44.385885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:44.385857 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" event={"ID":"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e","Type":"ContainerStarted","Data":"c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41"} Oct 30 06:30:45.392845 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:45.392762 6171 generic.go:356] "Generic (PLEG): container finished" podID="6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" containerID="7e3760eef9268de1e8ec1747ec1a02aa27467898bbe76c8d3500903769975d2d" exitCode=0 Oct 30 06:30:45.393254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:45.392843 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" event={"ID":"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e","Type":"ContainerDied","Data":"7e3760eef9268de1e8ec1747ec1a02aa27467898bbe76c8d3500903769975d2d"} Oct 30 06:30:46.542748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.542720 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:46.657686 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.657614 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\") pod \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " Oct 30 06:30:46.657875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.657729 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\" (UniqueName: \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\") pod \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " Oct 30 06:30:46.657875 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.657773 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\") pod \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\" (UID: \"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e\") " Oct 30 06:30:46.658211 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.658183 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca" pod "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" (UID: "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:46.659769 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.659741 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca" pod "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" (UID: "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:46.660018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.659996 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h" pod "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" (UID: "6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e"). InnerVolumeSpecName "kube-api-access-p749h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:46.766827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.759230 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p749h\" (UniqueName: \"kubernetes.io/projected/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-kube-api-access-p749h\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:46.766827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.759276 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:46.766827 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:46.759291 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:30:47.407186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:47.407159 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" Oct 30 06:30:47.407360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:47.407153 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2" event={"ID":"6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e","Type":"ContainerDied","Data":"c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41"} Oct 30 06:30:47.407360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:47.407301 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c73098509ec710bd6f0a18d03d035bdb1b89169a643df199223611741e2f8d41" Oct 30 06:30:54.743993 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:54.743955 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:30:56.379844 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:30:56.379812 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:30:59.581709 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:30:59.581675 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:04.637708 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.637628 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf"] Oct 30 06:31:04.638119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.638107 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" containerName="revoke-keys" Oct 30 06:31:04.638119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.638119 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" containerName="revoke-keys" Oct 30 06:31:04.638226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.638188 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" containerName="revoke-keys" Oct 30 06:31:04.641510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.641494 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.660679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.660651 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf"] Oct 30 06:31:04.672641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.672104 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:31:04.672641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.672111 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:31:04.672641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.672147 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:31:04.672641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.672634 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:31:04.691231 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.691198 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.691381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.691242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.691381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.691332 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6lxd\" (UniqueName: \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.792901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.792871 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6lxd\" (UniqueName: \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.793055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.793005 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.793055 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.793026 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.793846 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.793761 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.795533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.795511 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.819866 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.819836 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6lxd\" (UniqueName: \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\") pod \"maas-api-revoke-sub-0b7e8edb-pz4qf\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:04.955542 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:04.955475 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:05.261006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:05.260976 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf"] Oct 30 06:31:05.261780 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:31:05.261749 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24d44e9e_a3f1_4c2a_bd4a_182be7380819.slice/crio-8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143 WatchSource:0}: Error finding container 8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143: Status 404 returned error can't find the container with id 8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143 Oct 30 06:31:05.538965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:05.538881 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" event={"ID":"24d44e9e-a3f1-4c2a-bd4a-182be7380819","Type":"ContainerStarted","Data":"8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143"} Oct 30 06:31:06.546373 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:06.546331 6171 generic.go:356] "Generic (PLEG): container finished" podID="24d44e9e-a3f1-4c2a-bd4a-182be7380819" containerID="f0f35b2accd81576570a9881a4f7873125d887aa4884bc89e7026f8fe6762a4c" exitCode=0 Oct 30 06:31:06.546774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:06.546383 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" event={"ID":"24d44e9e-a3f1-4c2a-bd4a-182be7380819","Type":"ContainerDied","Data":"f0f35b2accd81576570a9881a4f7873125d887aa4884bc89e7026f8fe6762a4c"} Oct 30 06:31:07.696388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.696363 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:07.719533 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.719505 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\" (UniqueName: \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\") pod \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " Oct 30 06:31:07.719685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.719610 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\") pod \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " Oct 30 06:31:07.719685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.719655 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\") pod \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\" (UID: \"24d44e9e-a3f1-4c2a-bd4a-182be7380819\") " Oct 30 06:31:07.720010 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.719986 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca" pod "24d44e9e-a3f1-4c2a-bd4a-182be7380819" (UID: "24d44e9e-a3f1-4c2a-bd4a-182be7380819"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:07.721817 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.721784 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca" pod "24d44e9e-a3f1-4c2a-bd4a-182be7380819" (UID: "24d44e9e-a3f1-4c2a-bd4a-182be7380819"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:07.722113 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.721839 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd" pod "24d44e9e-a3f1-4c2a-bd4a-182be7380819" (UID: "24d44e9e-a3f1-4c2a-bd4a-182be7380819"). InnerVolumeSpecName "kube-api-access-d6lxd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:07.820697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.820613 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6lxd\" (UniqueName: \"kubernetes.io/projected/24d44e9e-a3f1-4c2a-bd4a-182be7380819-kube-api-access-d6lxd\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:07.820697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.820642 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:07.820697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:07.820652 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/24d44e9e-a3f1-4c2a-bd4a-182be7380819-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:08.561922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:08.561892 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" Oct 30 06:31:08.561922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:08.561914 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf" event={"ID":"24d44e9e-a3f1-4c2a-bd4a-182be7380819","Type":"ContainerDied","Data":"8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143"} Oct 30 06:31:08.562168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:08.561964 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a480398911911fe75a154cae1574edc155479967715b070b318134c6f31c143" Oct 30 06:31:15.429430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.429393 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9"] Oct 30 06:31:15.429857 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.429845 6171 cpu_manager.go:414] "containerMap: removing container" podUID="24d44e9e-a3f1-4c2a-bd4a-182be7380819" containerName="revoke-keys" Oct 30 06:31:15.429911 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.429858 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24d44e9e-a3f1-4c2a-bd4a-182be7380819" containerName="revoke-keys" Oct 30 06:31:15.429956 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.429935 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="24d44e9e-a3f1-4c2a-bd4a-182be7380819" containerName="revoke-keys" Oct 30 06:31:15.433171 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.433153 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.452143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.452117 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9"] Oct 30 06:31:15.463378 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.463345 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:31:15.463527 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.463414 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:31:15.463646 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.463628 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:31:15.464295 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.464274 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:31:15.494174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.494143 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.494352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.494209 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stdsc\" (UniqueName: \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.494352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.494300 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.594983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.594952 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.595170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.595034 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.595170 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.595126 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stdsc\" (UniqueName: \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.595681 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.595655 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.597508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.597482 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.620885 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.620857 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stdsc\" (UniqueName: \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\") pod \"maas-api-revoke-sub-3d4156f1-pgmj9\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.749510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.749472 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:15.864106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:15.864074 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:16.055012 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:16.054970 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9"] Oct 30 06:31:16.064440 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:31:16.064410 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e5cd4e_e0d8_401b_8b40_3b3ef4cb5675.slice/crio-d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5 WatchSource:0}: Error finding container d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5: Status 404 returned error can't find the container with id d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5 Oct 30 06:31:16.621083 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:16.621029 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" event={"ID":"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675","Type":"ContainerStarted","Data":"bf16b7523f5ec755145b012e38683a31a20c3766ccba2b6fd91f250d36c1bec0"} Oct 30 06:31:16.621523 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:16.621497 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" event={"ID":"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675","Type":"ContainerStarted","Data":"d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5"} Oct 30 06:31:17.629313 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:17.629272 6171 generic.go:356] "Generic (PLEG): container finished" podID="d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" containerID="bf16b7523f5ec755145b012e38683a31a20c3766ccba2b6fd91f250d36c1bec0" exitCode=0 Oct 30 06:31:17.629777 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:17.629356 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" event={"ID":"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675","Type":"ContainerDied","Data":"bf16b7523f5ec755145b012e38683a31a20c3766ccba2b6fd91f250d36c1bec0"} Oct 30 06:31:18.771035 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.771011 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:18.828325 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.828299 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\") pod \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " Oct 30 06:31:18.828481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.828373 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\" (UniqueName: \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\") pod \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " Oct 30 06:31:18.828481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.828438 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\") pod \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\" (UID: \"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675\") " Oct 30 06:31:18.828796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.828770 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca" pod "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" (UID: "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:18.830460 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.830439 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca" pod "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" (UID: "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:18.830565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.830543 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc" pod "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" (UID: "d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675"). InnerVolumeSpecName "kube-api-access-stdsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:18.929830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.929755 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:18.929830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.929792 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stdsc\" (UniqueName: \"kubernetes.io/projected/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-kube-api-access-stdsc\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:18.929830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:18.929807 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:19.642252 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:19.642222 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" Oct 30 06:31:19.642434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:19.642260 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9" event={"ID":"d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675","Type":"ContainerDied","Data":"d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5"} Oct 30 06:31:19.642434 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:19.642312 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d824ba32fa9d95c1f0f0bcf9ace56922f0a62355d73dc246f8f1edbebe6612b5" Oct 30 06:31:26.379993 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:31:26.379942 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:31:26.631352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:26.631270 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:31.362587 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:31.362558 6171 scope.go:122] "RemoveContainer" containerID="d2356354aa5edf9e3e7fe7f7f11c1be184f7cd837c09f04dd74cda220d30a878" Oct 30 06:31:31.372015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:31.371990 6171 scope.go:122] "RemoveContainer" containerID="2b3ff08a593016ef903a4a7146988447cfb21d8a19477fdbd6547b04a2b7f8ee" Oct 30 06:31:31.381023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:31.381005 6171 scope.go:122] "RemoveContainer" containerID="f5a7517187eadc2995fbb20806ccfdf5aedd7645d8a8b6bc167e3abdfe46ff36" Oct 30 06:31:31.613160 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:31.613083 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:36.751863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.751828 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9"] Oct 30 06:31:36.767232 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.767205 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-pgmj9"] Oct 30 06:31:36.799661 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.799623 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn"] Oct 30 06:31:36.800134 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.800119 6171 cpu_manager.go:414] "containerMap: removing container" podUID="d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" containerName="revoke-keys" Oct 30 06:31:36.800193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.800135 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" containerName="revoke-keys" Oct 30 06:31:36.800242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.800214 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" containerName="revoke-keys" Oct 30 06:31:36.805203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.805183 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:36.823366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.823328 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:31:36.823366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.823348 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:31:36.823611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.823410 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:31:36.823611 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.823486 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:31:36.823793 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.823770 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn"] Oct 30 06:31:36.939553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.939517 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:36.939553 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.939557 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:36.939839 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:36.939763 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8blz9\" (UniqueName: \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.040795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.040716 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.040795 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.040753 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.040994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.040833 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8blz9\" (UniqueName: \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.041524 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.041501 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.043296 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.043276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.066551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.066524 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8blz9\" (UniqueName: \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\") pod \"maas-api-revoke-sub-3d4156f1-d6xdn\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.119644 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.119615 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:37.423314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.423284 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn"] Oct 30 06:31:37.424512 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:31:37.424483 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f791f06_afc5_4ce1_9d56_7776c37446c2.slice/crio-bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3 WatchSource:0}: Error finding container bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3: Status 404 returned error can't find the container with id bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3 Oct 30 06:31:37.670262 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.670223 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675" path="/var/lib/kubelet/pods/d0e5cd4e-e0d8-401b-8b40-3b3ef4cb5675/volumes" Oct 30 06:31:37.837389 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.837347 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" event={"ID":"0f791f06-afc5-4ce1-9d56-7776c37446c2","Type":"ContainerStarted","Data":"84410bd273d9c013b7f1e2f2762582ee3b74bc0b83af1885934451bf000adc29"} Oct 30 06:31:37.837799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.837410 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" event={"ID":"0f791f06-afc5-4ce1-9d56-7776c37446c2","Type":"ContainerStarted","Data":"bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3"} Oct 30 06:31:37.883352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:37.883307 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" podStartSLOduration=1.883293982 podStartE2EDuration="1.883293982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:37.881134251 +0000 UTC m=+3556.321511464" watchObservedRunningTime="2026-10-30 06:31:37.883293982 +0000 UTC m=+3556.323671175" Oct 30 06:31:38.844002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:38.843967 6171 generic.go:356] "Generic (PLEG): container finished" podID="0f791f06-afc5-4ce1-9d56-7776c37446c2" containerID="84410bd273d9c013b7f1e2f2762582ee3b74bc0b83af1885934451bf000adc29" exitCode=0 Oct 30 06:31:38.844429 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:38.844007 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" event={"ID":"0f791f06-afc5-4ce1-9d56-7776c37446c2","Type":"ContainerDied","Data":"84410bd273d9c013b7f1e2f2762582ee3b74bc0b83af1885934451bf000adc29"} Oct 30 06:31:39.989023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:39.988997 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:40.067998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.067960 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\") pod \"0f791f06-afc5-4ce1-9d56-7776c37446c2\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " Oct 30 06:31:40.067998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.068004 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\" (UniqueName: \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\") pod \"0f791f06-afc5-4ce1-9d56-7776c37446c2\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " Oct 30 06:31:40.068286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.068117 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\") pod \"0f791f06-afc5-4ce1-9d56-7776c37446c2\" (UID: \"0f791f06-afc5-4ce1-9d56-7776c37446c2\") " Oct 30 06:31:40.068392 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.068357 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca" pod "0f791f06-afc5-4ce1-9d56-7776c37446c2" (UID: "0f791f06-afc5-4ce1-9d56-7776c37446c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:40.070371 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.070342 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca" pod "0f791f06-afc5-4ce1-9d56-7776c37446c2" (UID: "0f791f06-afc5-4ce1-9d56-7776c37446c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:40.070543 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.070513 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9" pod "0f791f06-afc5-4ce1-9d56-7776c37446c2" (UID: "0f791f06-afc5-4ce1-9d56-7776c37446c2"). InnerVolumeSpecName "kube-api-access-8blz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:40.169923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.169834 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:40.169923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.169866 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f791f06-afc5-4ce1-9d56-7776c37446c2-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:40.169923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.169877 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8blz9\" (UniqueName: \"kubernetes.io/projected/0f791f06-afc5-4ce1-9d56-7776c37446c2-kube-api-access-8blz9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:31:40.856743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.856711 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" event={"ID":"0f791f06-afc5-4ce1-9d56-7776c37446c2","Type":"ContainerDied","Data":"bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3"} Oct 30 06:31:40.856965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.856766 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc3dc4c820724b97c1765248e4bff252cb7d6739d8dbdc7083805eadf8115ff3" Oct 30 06:31:40.856965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:40.856728 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn" Oct 30 06:31:48.754775 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:48.754736 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:49.034892 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:49.034818 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2"] Oct 30 06:31:49.048836 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:49.048808 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-35f05329-9mcg2"] Oct 30 06:31:49.670671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:49.670636 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3eb8707-8a10-4e03-812a-72b490acb187" path="/var/lib/kubelet/pods/a3eb8707-8a10-4e03-812a-72b490acb187/volumes" Oct 30 06:31:53.767609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:31:53.767556 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:31:56.379504 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:31:56.379470 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:32:01.095253 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.095217 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn"] Oct 30 06:32:01.108957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.108931 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-d6xdn"] Oct 30 06:32:01.137641 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.137613 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs"] Oct 30 06:32:01.138087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.138057 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0f791f06-afc5-4ce1-9d56-7776c37446c2" containerName="revoke-keys" Oct 30 06:32:01.138087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.138075 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f791f06-afc5-4ce1-9d56-7776c37446c2" containerName="revoke-keys" Oct 30 06:32:01.138302 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.138187 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f791f06-afc5-4ce1-9d56-7776c37446c2" containerName="revoke-keys" Oct 30 06:32:01.141570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.141552 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.158724 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.158698 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:32:01.159393 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.159352 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:01.159544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.159400 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:01.159544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.159440 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:32:01.162873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.162842 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs"] Oct 30 06:32:01.172938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.172916 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.173103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.172962 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zr9\" (UniqueName: \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.173103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.172999 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.274471 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.274435 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.274671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.274507 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zr9\" (UniqueName: \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.274671 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.274551 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.275175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.275153 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.276940 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.276918 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.300298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.300257 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zr9\" (UniqueName: \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\") pod \"maas-api-revoke-sub-3d4156f1-9xczs\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.454762 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.454685 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:01.670616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.670582 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f791f06-afc5-4ce1-9d56-7776c37446c2" path="/var/lib/kubelet/pods/0f791f06-afc5-4ce1-9d56-7776c37446c2/volumes" Oct 30 06:32:01.755305 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.755274 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs"] Oct 30 06:32:01.756742 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:32:01.756712 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65cc98fb_539c_488e_b591_412e32c43d12.slice/crio-ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1 WatchSource:0}: Error finding container ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1: Status 404 returned error can't find the container with id ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1 Oct 30 06:32:01.990112 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:01.990076 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" event={"ID":"65cc98fb-539c-488e-b591-412e32c43d12","Type":"ContainerStarted","Data":"ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1"} Oct 30 06:32:02.997386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:02.997346 6171 generic.go:356] "Generic (PLEG): container finished" podID="65cc98fb-539c-488e-b591-412e32c43d12" containerID="6f7e3ad902b4c7335add2229f938861930f932bd9440d986ec83c959bd462731" exitCode=0 Oct 30 06:32:02.997824 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:02.997428 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" event={"ID":"65cc98fb-539c-488e-b591-412e32c43d12","Type":"ContainerDied","Data":"6f7e3ad902b4c7335add2229f938861930f932bd9440d986ec83c959bd462731"} Oct 30 06:32:04.148085 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.148058 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:04.200719 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.200688 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\") pod \"65cc98fb-539c-488e-b591-412e32c43d12\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " Oct 30 06:32:04.200926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.200751 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\") pod \"65cc98fb-539c-488e-b591-412e32c43d12\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " Oct 30 06:32:04.200926 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.200792 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\" (UniqueName: \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\") pod \"65cc98fb-539c-488e-b591-412e32c43d12\" (UID: \"65cc98fb-539c-488e-b591-412e32c43d12\") " Oct 30 06:32:04.201432 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.201405 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca" pod "65cc98fb-539c-488e-b591-412e32c43d12" (UID: "65cc98fb-539c-488e-b591-412e32c43d12"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:32:04.203099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.203071 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca" pod "65cc98fb-539c-488e-b591-412e32c43d12" (UID: "65cc98fb-539c-488e-b591-412e32c43d12"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:32:04.203186 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.203105 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9" pod "65cc98fb-539c-488e-b591-412e32c43d12" (UID: "65cc98fb-539c-488e-b591-412e32c43d12"). InnerVolumeSpecName "kube-api-access-75zr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:32:04.301869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.301783 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:32:04.301869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.301821 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/65cc98fb-539c-488e-b591-412e32c43d12-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:32:04.301869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:04.301832 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zr9\" (UniqueName: \"kubernetes.io/projected/65cc98fb-539c-488e-b591-412e32c43d12-kube-api-access-75zr9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:32:05.011942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:05.011909 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" Oct 30 06:32:05.011942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:05.011927 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs" event={"ID":"65cc98fb-539c-488e-b591-412e32c43d12","Type":"ContainerDied","Data":"ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1"} Oct 30 06:32:05.012207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:05.011979 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ebc84674a547823c544990a22efbb94b7a24597798f77233c61b97ceba8300c1" Oct 30 06:32:13.044021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:13.043987 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:32:18.115864 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:18.115828 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:32:26.381265 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:32:26.381223 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:32:27.987022 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:27.986985 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:32:27.987485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:27.987467 6171 cpu_manager.go:414] "containerMap: removing container" podUID="65cc98fb-539c-488e-b591-412e32c43d12" containerName="revoke-keys" Oct 30 06:32:27.987528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:27.987487 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cc98fb-539c-488e-b591-412e32c43d12" containerName="revoke-keys" Oct 30 06:32:27.987596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:27.987588 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cc98fb-539c-488e-b591-412e32c43d12" containerName="revoke-keys" Oct 30 06:32:27.991037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:27.991015 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.008809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.008780 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:28.008994 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.008934 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:28.023136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.023111 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-6c0c09-serving-cert\"" Oct 30 06:32:28.023282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.023196 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4vtg8\"" Oct 30 06:32:28.023622 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.023598 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:32:28.027496 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.027467 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:32:28.122374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.122342 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.122567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.122404 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf265\" (UniqueName: \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.122567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.122482 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.122567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.122525 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.223188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.223144 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.223188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.223257 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf265\" (UniqueName: \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.223188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.223292 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.223188 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.223328 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.230612 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.227276 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.240080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.239973 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.243660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.243591 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.263984 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.263955 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf265\" (UniqueName: \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\") pod \"maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.308834 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.308806 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:28.625737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.625146 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:32:28.681799 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.681733 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:32:28.691164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.691139 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.705345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.705315 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:32:28.708287 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.708262 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-73b62168-dockercfg-bw9gt\"" Oct 30 06:32:28.708420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.708368 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-6c0c09\"" Oct 30 06:32:28.829120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.829080 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.829120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.829122 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dr8d\" (UniqueName: \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.930375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.930290 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dr8d\" (UniqueName: \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.930547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.930434 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.931144 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.931102 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:28.960520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:28.960493 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dr8d\" (UniqueName: \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\") pod \"payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:29.013099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.013034 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:29.156676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.156647 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" event={"ID":"b91683d4-fe98-4860-9853-37046aed6a85","Type":"ContainerStarted","Data":"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2"} Oct 30 06:32:29.156813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.156703 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" event={"ID":"b91683d4-fe98-4860-9853-37046aed6a85","Type":"ContainerStarted","Data":"5a694db3069158817f770a853a726e13a7ad19fcfa9ffe17dc3454c0adabf7eb"} Oct 30 06:32:29.156886 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.156872 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:29.203831 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.203748 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" podStartSLOduration=1.788726036 podStartE2EDuration="2.203735979s" totalImagesPullingTime="415.009943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:29.202557171 +0000 UTC m=+3607.642934401" watchObservedRunningTime="2026-10-30 06:32:29.203735979 +0000 UTC m=+3607.644113176" Oct 30 06:32:29.341254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:29.341208 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:32:30.164039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:30.164005 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" event={"ID":"4aca69cb-d23e-4281-92a5-052a1d4394ce","Type":"ContainerStarted","Data":"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e"} Oct 30 06:32:30.164487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:30.164068 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" event={"ID":"4aca69cb-d23e-4281-92a5-052a1d4394ce","Type":"ContainerStarted","Data":"c99a7df4bd0baef563001fda29b1318049c244e69103921c3c5e1d5f169a9b3e"} Oct 30 06:32:30.164487 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:30.164095 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:30.208036 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:30.207992 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" podStartSLOduration=2.207979441 podStartE2EDuration="2.207979441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:32:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:32:30.207800495 +0000 UTC m=+3608.648177688" watchObservedRunningTime="2026-10-30 06:32:30.207979441 +0000 UTC m=+3608.648356633" Oct 30 06:32:31.412386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:31.412299 6171 scope.go:122] "RemoveContainer" containerID="9488ff406f9e3a795f1ca3882267f3c4010ea154409fb8beaff3b9d4b29365d7" Oct 30 06:32:31.422098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:31.422078 6171 scope.go:122] "RemoveContainer" containerID="ed478e2e01b521ed049f72962d5f46d56719711a6fb7e8ed84e4735c14f9906c" Oct 30 06:32:31.430509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:31.430489 6171 scope.go:122] "RemoveContainer" containerID="5347dd7ba6391055752a23127fde2c2a8b728eced2f05aa919808c2f7acd00cd" Oct 30 06:32:31.439583 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:31.439563 6171 scope.go:122] "RemoveContainer" containerID="dff3eb970f32a788e24e92f85f80561309775570b9d8f1d7c86fc1afc837aa84" Oct 30 06:32:31.740516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:31.740480 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:32:35.169574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:35.169542 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:32:41.031997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:41.031954 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2"] Oct 30 06:32:41.046109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:41.046079 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-92lt2"] Oct 30 06:32:41.172805 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:41.172779 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:32:41.673021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:41.672985 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb87ca31-5767-465b-8c4a-239f95c24403" path="/var/lib/kubelet/pods/cb87ca31-5767-465b-8c4a-239f95c24403/volumes" Oct 30 06:32:47.043962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:32:47.043925 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:32:56.379666 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:32:56.379629 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:33:01.420747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:01.420702 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:33:18.760597 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:18.760560 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:33:26.381023 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:33:26.380979 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:33:29.054491 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:29.054448 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:33:30.096422 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.096382 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:33:30.097016 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.096635 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" podUID="b91683d4-fe98-4860-9853-37046aed6a85" containerName="maas-api" containerID="cri-o://fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2" gracePeriod=30 Oct 30 06:33:30.165196 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.165161 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" podUID="b91683d4-fe98-4860-9853-37046aed6a85" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.205:8443/health\": dial tcp 10.133.1.205:8443: connect: connection refused" Oct 30 06:33:30.371189 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.371167 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:33:30.448368 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.448324 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\") pod \"b91683d4-fe98-4860-9853-37046aed6a85\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " Oct 30 06:33:30.448545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.448414 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\") pod \"b91683d4-fe98-4860-9853-37046aed6a85\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " Oct 30 06:33:30.448545 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.448516 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\") pod \"b91683d4-fe98-4860-9853-37046aed6a85\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " Oct 30 06:33:30.448633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.448593 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\" (UniqueName: \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\") pod \"b91683d4-fe98-4860-9853-37046aed6a85\" (UID: \"b91683d4-fe98-4860-9853-37046aed6a85\") " Oct 30 06:33:30.448987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.448956 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca" pod "b91683d4-fe98-4860-9853-37046aed6a85" (UID: "b91683d4-fe98-4860-9853-37046aed6a85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:30.450783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.450754 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs" pod "b91683d4-fe98-4860-9853-37046aed6a85" (UID: "b91683d4-fe98-4860-9853-37046aed6a85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:30.451476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.451453 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls" pod "b91683d4-fe98-4860-9853-37046aed6a85" (UID: "b91683d4-fe98-4860-9853-37046aed6a85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:30.451564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.451492 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265" pod "b91683d4-fe98-4860-9853-37046aed6a85" (UID: "b91683d4-fe98-4860-9853-37046aed6a85"). InnerVolumeSpecName "kube-api-access-rf265". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:30.543465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.543424 6171 generic.go:356] "Generic (PLEG): container finished" podID="b91683d4-fe98-4860-9853-37046aed6a85" containerID="fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2" exitCode=0 Oct 30 06:33:30.543682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.543548 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" Oct 30 06:33:30.543682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.543651 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" event={"ID":"b91683d4-fe98-4860-9853-37046aed6a85","Type":"ContainerDied","Data":"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2"} Oct 30 06:33:30.543815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.543695 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt" event={"ID":"b91683d4-fe98-4860-9853-37046aed6a85","Type":"ContainerDied","Data":"5a694db3069158817f770a853a726e13a7ad19fcfa9ffe17dc3454c0adabf7eb"} Oct 30 06:33:30.543815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.543757 6171 scope.go:122] "RemoveContainer" containerID="fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2" Oct 30 06:33:30.552778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.549722 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:30.552778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.549753 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b91683d4-fe98-4860-9853-37046aed6a85-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:30.552778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.549768 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b91683d4-fe98-4860-9853-37046aed6a85-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:30.552778 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.549782 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf265\" (UniqueName: \"kubernetes.io/projected/b91683d4-fe98-4860-9853-37046aed6a85-kube-api-access-rf265\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:30.566254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.566222 6171 scope.go:122] "RemoveContainer" containerID="fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2" Oct 30 06:33:30.566675 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:33:30.566649 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2\": container with ID starting with fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2 not found: ID does not exist" containerID="fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2" Oct 30 06:33:30.566801 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.566687 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2"} err="failed to get container status \"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2\": rpc error: code = NotFound desc = could not find container \"fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2\": container with ID starting with fbb200d5af7ef18d99f648a945038991a8b05d3ac4865507a8ac3bf58539e0a2 not found: ID does not exist" Oct 30 06:33:30.607944 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.607858 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:33:30.611783 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:30.611754 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-6c0c09-57df7b6c56-k2dqt"] Oct 30 06:33:31.499168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:31.499133 6171 scope.go:122] "RemoveContainer" containerID="a682a26cd01ec4c671f3dcbc5ad3d47c34abcc5b014eeb3271f792c50c5bfe51" Oct 30 06:33:31.508829 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:31.508802 6171 scope.go:122] "RemoveContainer" containerID="717b1a87b1cbd13499f040ce0ea0cf22c6b4ba01bd8c110c2c9e2d4545ac904f" Oct 30 06:33:31.670733 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:31.670704 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91683d4-fe98-4860-9853-37046aed6a85" path="/var/lib/kubelet/pods/b91683d4-fe98-4860-9853-37046aed6a85/volumes" Oct 30 06:33:32.029998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:32.029964 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv"] Oct 30 06:33:32.044168 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:32.044134 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-kq2vv"] Oct 30 06:33:32.263908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:32.263867 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:33:33.669792 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:33.669755 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09cd97c4-9e6e-4ee7-b177-ba2f21e32725" path="/var/lib/kubelet/pods/09cd97c4-9e6e-4ee7-b177-ba2f21e32725/volumes" Oct 30 06:33:34.054970 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:34.054912 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" containerID="cri-o://f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e" gracePeriod=25 Oct 30 06:33:39.410407 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.410367 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs"] Oct 30 06:33:39.424366 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.424327 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9xczs"] Oct 30 06:33:39.454630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.454607 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz"] Oct 30 06:33:39.455069 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.455056 6171 cpu_manager.go:414] "containerMap: removing container" podUID="b91683d4-fe98-4860-9853-37046aed6a85" containerName="maas-api" Oct 30 06:33:39.455132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.455070 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91683d4-fe98-4860-9853-37046aed6a85" containerName="maas-api" Oct 30 06:33:39.455174 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.455163 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91683d4-fe98-4860-9853-37046aed6a85" containerName="maas-api" Oct 30 06:33:39.460213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.460194 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.478966 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.478935 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz"] Oct 30 06:33:39.479284 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.479255 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:33:39.479419 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.479368 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:33:39.495973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.495937 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:33:39.496128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.495961 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:33:39.556032 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.556003 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.556204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.556133 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.556279 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.556242 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9txr\" (UniqueName: \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.657298 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.657266 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9txr\" (UniqueName: \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.657433 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.657304 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.657528 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.657510 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.658151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.658069 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.659737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.659716 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.672058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.671992 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cc98fb-539c-488e-b591-412e32c43d12" path="/var/lib/kubelet/pods/65cc98fb-539c-488e-b591-412e32c43d12/volumes" Oct 30 06:33:39.682235 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.682209 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9txr\" (UniqueName: \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\") pod \"maas-api-revoke-sub-3d4156f1-qd9fz\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:39.774345 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:39.774317 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:40.074307 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:40.074272 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz"] Oct 30 06:33:40.075452 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:33:40.075426 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a077f5f_994c_4474_af7f_45baf9bf3430.slice/crio-b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b WatchSource:0}: Error finding container b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b: Status 404 returned error can't find the container with id b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b Oct 30 06:33:40.631585 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:40.631549 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" event={"ID":"3a077f5f-994c-4474-af7f-45baf9bf3430","Type":"ContainerStarted","Data":"204cdcf6b60e08e16d64af0eea1322bdcaf35fe2a37c794789aa3c8fa0c352e1"} Oct 30 06:33:40.631973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:40.631605 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" event={"ID":"3a077f5f-994c-4474-af7f-45baf9bf3430","Type":"ContainerStarted","Data":"b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b"} Oct 30 06:33:40.690510 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:40.690427 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" podStartSLOduration=1.690413694 podStartE2EDuration="1.690413694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:40.674733524 +0000 UTC m=+3679.115110719" watchObservedRunningTime="2026-10-30 06:33:40.690413694 +0000 UTC m=+3679.130790939" Oct 30 06:33:41.171734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:41.171704 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:33:41.639011 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:41.638977 6171 generic.go:356] "Generic (PLEG): container finished" podID="3a077f5f-994c-4474-af7f-45baf9bf3430" containerID="204cdcf6b60e08e16d64af0eea1322bdcaf35fe2a37c794789aa3c8fa0c352e1" exitCode=0 Oct 30 06:33:41.639465 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:41.639075 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" event={"ID":"3a077f5f-994c-4474-af7f-45baf9bf3430","Type":"ContainerDied","Data":"204cdcf6b60e08e16d64af0eea1322bdcaf35fe2a37c794789aa3c8fa0c352e1"} Oct 30 06:33:42.794278 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.794251 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:42.886658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.886630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\") pod \"3a077f5f-994c-4474-af7f-45baf9bf3430\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " Oct 30 06:33:42.886810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.886690 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\") pod \"3a077f5f-994c-4474-af7f-45baf9bf3430\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " Oct 30 06:33:42.886810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.886732 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\" (UniqueName: \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\") pod \"3a077f5f-994c-4474-af7f-45baf9bf3430\" (UID: \"3a077f5f-994c-4474-af7f-45baf9bf3430\") " Oct 30 06:33:42.887025 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.887000 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca" pod "3a077f5f-994c-4474-af7f-45baf9bf3430" (UID: "3a077f5f-994c-4474-af7f-45baf9bf3430"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:42.888906 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.888880 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca" pod "3a077f5f-994c-4474-af7f-45baf9bf3430" (UID: "3a077f5f-994c-4474-af7f-45baf9bf3430"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:42.889028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.888942 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr" pod "3a077f5f-994c-4474-af7f-45baf9bf3430" (UID: "3a077f5f-994c-4474-af7f-45baf9bf3430"). InnerVolumeSpecName "kube-api-access-h9txr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:42.987357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.987331 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:42.987357 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.987354 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a077f5f-994c-4474-af7f-45baf9bf3430-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:42.987520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:42.987364 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9txr\" (UniqueName: \"kubernetes.io/projected/3a077f5f-994c-4474-af7f-45baf9bf3430-kube-api-access-h9txr\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:43.651743 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:43.651715 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" Oct 30 06:33:43.652096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:43.651711 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz" event={"ID":"3a077f5f-994c-4474-af7f-45baf9bf3430","Type":"ContainerDied","Data":"b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b"} Oct 30 06:33:43.652096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:43.651865 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6e0bc88a3f13e5d32442b816d1b04cd67590bce5abc02798a9f2307e8d2189b" Oct 30 06:33:51.171947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:51.171906 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:33:51.537930 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:51.537891 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:33:52.033894 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:52.033852 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6"] Oct 30 06:33:52.049149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:52.049114 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-q5tb6"] Oct 30 06:33:53.697682 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:53.697479 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d03c1432-c498-4254-a936-e59d6a85d8d6" path="/var/lib/kubelet/pods/d03c1432-c498-4254-a936-e59d6a85d8d6/volumes" Oct 30 06:33:54.306986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.306959 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:33:54.438087 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.437931 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\") pod \"4aca69cb-d23e-4281-92a5-052a1d4394ce\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " Oct 30 06:33:54.438241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.438113 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\" (UniqueName: \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\") pod \"4aca69cb-d23e-4281-92a5-052a1d4394ce\" (UID: \"4aca69cb-d23e-4281-92a5-052a1d4394ce\") " Oct 30 06:33:54.438241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.438221 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume" pod "4aca69cb-d23e-4281-92a5-052a1d4394ce" (UID: "4aca69cb-d23e-4281-92a5-052a1d4394ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:54.438490 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.438472 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4aca69cb-d23e-4281-92a5-052a1d4394ce-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:54.440507 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.440490 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d" pod "4aca69cb-d23e-4281-92a5-052a1d4394ce" (UID: "4aca69cb-d23e-4281-92a5-052a1d4394ce"). InnerVolumeSpecName "kube-api-access-8dr8d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:54.539551 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.539517 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dr8d\" (UniqueName: \"kubernetes.io/projected/4aca69cb-d23e-4281-92a5-052a1d4394ce-kube-api-access-8dr8d\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:33:54.764467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.764434 6171 generic.go:356] "Generic (PLEG): container finished" podID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerID="f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e" exitCode=0 Oct 30 06:33:54.764891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.764493 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" Oct 30 06:33:54.764891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.764518 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" event={"ID":"4aca69cb-d23e-4281-92a5-052a1d4394ce","Type":"ContainerDied","Data":"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e"} Oct 30 06:33:54.764891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.764576 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz" event={"ID":"4aca69cb-d23e-4281-92a5-052a1d4394ce","Type":"ContainerDied","Data":"c99a7df4bd0baef563001fda29b1318049c244e69103921c3c5e1d5f169a9b3e"} Oct 30 06:33:54.764891 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.764605 6171 scope.go:122] "RemoveContainer" containerID="f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e" Oct 30 06:33:54.774633 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.774615 6171 scope.go:122] "RemoveContainer" containerID="f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e" Oct 30 06:33:54.774892 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:33:54.774868 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e\": container with ID starting with f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e not found: ID does not exist" containerID="f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e" Oct 30 06:33:54.774996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.774897 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e"} err="failed to get container status \"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e\": rpc error: code = NotFound desc = could not find container \"f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e\": container with ID starting with f910e25760b83678e606cb734f6a842e665206709d584689f14f2ea68c1daf4e not found: ID does not exist" Oct 30 06:33:54.823359 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.823333 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:33:54.828145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:54.828127 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-6c0c09-6797c446bxfmxz"] Oct 30 06:33:55.671388 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:55.671357 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" path="/var/lib/kubelet/pods/4aca69cb-d23e-4281-92a5-052a1d4394ce/volumes" Oct 30 06:33:56.379380 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:33:56.379344 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:33:57.019683 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:33:57.019651 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:34:02.031428 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:02.031351 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8"] Oct 30 06:34:02.046662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:02.046623 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-h7nh8"] Oct 30 06:34:03.671299 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:03.671266 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb" path="/var/lib/kubelet/pods/1952b7b0-aeb3-45a5-9f95-a2dc7c27b1bb/volumes" Oct 30 06:34:26.380104 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:34:26.380069 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:34:31.029442 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:31.029405 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr"] Oct 30 06:34:31.043327 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:31.043297 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-5xrlr"] Oct 30 06:34:31.540086 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:31.540040 6171 scope.go:122] "RemoveContainer" containerID="fcf366cbbd7bafba4d16faa83708333f37a0c9e66a518de37a8bfc96ee1ff6d5" Oct 30 06:34:31.550019 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:31.550000 6171 scope.go:122] "RemoveContainer" containerID="62f957ce6da6e71013de61271ab507d04eeed161f1716c0b68859eaf85ec94fd" Oct 30 06:34:31.670318 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:31.670289 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09577c80-9236-4833-b849-0fd9c04d833f" path="/var/lib/kubelet/pods/09577c80-9236-4833-b849-0fd9c04d833f/volumes" Oct 30 06:34:41.031319 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:41.031288 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79"] Oct 30 06:34:41.044234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:41.044210 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-8wm79"] Oct 30 06:34:41.678616 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:34:41.678578 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc70a70e-b406-4277-b22d-7181efc69d21" path="/var/lib/kubelet/pods/dc70a70e-b406-4277-b22d-7181efc69d21/volumes" Oct 30 06:34:56.380481 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:34:56.380443 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:35:04.029665 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:04.029627 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx"] Oct 30 06:35:04.043558 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:04.043529 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-kwqmx"] Oct 30 06:35:05.670387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:05.670344 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1789d1-2b91-4faf-93cb-ce615879d701" path="/var/lib/kubelet/pods/3d1789d1-2b91-4faf-93cb-ce615879d701/volumes" Oct 30 06:35:23.855576 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:23.855544 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:35:25.099588 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:25.099548 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4"] Oct 30 06:35:25.113723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:25.113674 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-ljcw4"] Oct 30 06:35:25.694761 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:25.694722 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f" path="/var/lib/kubelet/pods/f5ffcd6d-3fb2-4b77-966a-aa5e1a96914f/volumes" Oct 30 06:35:26.559074 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:35:26.557300 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:35:27.118207 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.118163 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-84wh7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:27.118420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.118241 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-84wh7" podUID="b3cabe1b-3080-4d86-b821-a1e469c7439e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:27.861385 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861353 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h"] Oct 30 06:35:27.861796 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861782 6171 cpu_manager.go:414] "containerMap: removing container" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" Oct 30 06:35:27.861848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861798 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" Oct 30 06:35:27.861848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861811 6171 cpu_manager.go:414] "containerMap: removing container" podUID="3a077f5f-994c-4474-af7f-45baf9bf3430" containerName="revoke-keys" Oct 30 06:35:27.861848 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861816 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a077f5f-994c-4474-af7f-45baf9bf3430" containerName="revoke-keys" Oct 30 06:35:27.861948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861897 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a077f5f-994c-4474-af7f-45baf9bf3430" containerName="revoke-keys" Oct 30 06:35:27.861948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.861912 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aca69cb-d23e-4281-92a5-052a1d4394ce" containerName="payload-processing" Oct 30 06:35:27.864791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.864776 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:27.882380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.882356 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:27.896582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.896554 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:27.896582 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.896574 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:27.896747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.896589 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:35:27.899030 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:27.899010 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h"] Oct 30 06:35:28.035247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.035218 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.035382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.035301 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.035467 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.035449 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt8x5\" (UniqueName: \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.136481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.136417 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt8x5\" (UniqueName: \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.136481 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.136461 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.136662 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.136486 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.137106 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.137082 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.138907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.138881 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.161667 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.161642 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt8x5\" (UniqueName: \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\") pod \"maas-api-revoke-sub-d576e68b-p985h\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.177613 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.177591 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:28.481701 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:28.481674 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h"] Oct 30 06:35:28.482789 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:35:28.482765 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113d462d_8e7f_4bc9_8cb2_5d4f24028098.slice/crio-bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada WatchSource:0}: Error finding container bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada: Status 404 returned error can't find the container with id bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada Oct 30 06:35:29.451579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:29.451545 6171 generic.go:356] "Generic (PLEG): container finished" podID="113d462d-8e7f-4bc9-8cb2-5d4f24028098" containerID="d979dbd83a105eb629bf3a901bde32013bd3cb940dd678cf79fb7137b93828aa" exitCode=0 Oct 30 06:35:29.451937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:29.451616 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" event={"ID":"113d462d-8e7f-4bc9-8cb2-5d4f24028098","Type":"ContainerDied","Data":"d979dbd83a105eb629bf3a901bde32013bd3cb940dd678cf79fb7137b93828aa"} Oct 30 06:35:29.451937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:29.451676 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" event={"ID":"113d462d-8e7f-4bc9-8cb2-5d4f24028098","Type":"ContainerStarted","Data":"bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada"} Oct 30 06:35:30.602309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.602285 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:30.761637 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.761610 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\") pod \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " Oct 30 06:35:30.761789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.761752 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\") pod \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " Oct 30 06:35:30.761849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.761814 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\" (UniqueName: \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\") pod \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\" (UID: \"113d462d-8e7f-4bc9-8cb2-5d4f24028098\") " Oct 30 06:35:30.762130 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.762106 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca" pod "113d462d-8e7f-4bc9-8cb2-5d4f24028098" (UID: "113d462d-8e7f-4bc9-8cb2-5d4f24028098"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:30.763828 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.763806 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca" pod "113d462d-8e7f-4bc9-8cb2-5d4f24028098" (UID: "113d462d-8e7f-4bc9-8cb2-5d4f24028098"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:30.763996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.763979 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5" pod "113d462d-8e7f-4bc9-8cb2-5d4f24028098" (UID: "113d462d-8e7f-4bc9-8cb2-5d4f24028098"). InnerVolumeSpecName "kube-api-access-jt8x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:30.863447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.863418 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:30.863447 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.863444 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jt8x5\" (UniqueName: \"kubernetes.io/projected/113d462d-8e7f-4bc9-8cb2-5d4f24028098-kube-api-access-jt8x5\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:30.863638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:30.863454 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/113d462d-8e7f-4bc9-8cb2-5d4f24028098-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:31.464509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.464438 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" event={"ID":"113d462d-8e7f-4bc9-8cb2-5d4f24028098","Type":"ContainerDied","Data":"bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada"} Oct 30 06:35:31.464509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.464455 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h" Oct 30 06:35:31.464509 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.464502 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcbaff1d015ddfddbedbd020328024802082248656ac0320bf240e8332185ada" Oct 30 06:35:31.583649 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.583620 6171 scope.go:122] "RemoveContainer" containerID="fdaf24209445ab7b7ea8241f6f0218cee093772bab1de6f35fda57c726d7ba6d" Oct 30 06:35:31.592692 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.592674 6171 scope.go:122] "RemoveContainer" containerID="4b4e6b8e1b083238ee6072964f0b40c1906402798a9e434a0ff00f11d442813b" Oct 30 06:35:31.601425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:31.601404 6171 scope.go:122] "RemoveContainer" containerID="5ccfed1ec3dd268a5ac457f7c8fd00e8fc6e50c469fa4930946de46f63f782df" Oct 30 06:35:39.026942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:39.026908 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:35:44.282565 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.282526 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj"] Oct 30 06:35:44.283813 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.283783 6171 cpu_manager.go:414] "containerMap: removing container" podUID="113d462d-8e7f-4bc9-8cb2-5d4f24028098" containerName="revoke-keys" Oct 30 06:35:44.283986 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.283971 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="113d462d-8e7f-4bc9-8cb2-5d4f24028098" containerName="revoke-keys" Oct 30 06:35:44.284320 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.284302 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="113d462d-8e7f-4bc9-8cb2-5d4f24028098" containerName="revoke-keys" Oct 30 06:35:44.303464 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.303426 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.313547 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.313519 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj"] Oct 30 06:35:44.327714 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.327691 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:44.327998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.327978 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:44.341028 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.341004 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:44.341381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.341362 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:35:44.428194 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.428158 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.428412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.428230 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.428412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.428341 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64md2\" (UniqueName: \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.529820 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.529786 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.529973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.529860 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.529973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.529893 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64md2\" (UniqueName: \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.530520 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.530496 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.532341 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.532316 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.555508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.555453 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64md2\" (UniqueName: \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\") pod \"maas-api-revoke-sub-d2bc93bc-4lcmj\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.629773 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.629749 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:44.932666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:44.929992 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj"] Oct 30 06:35:45.590479 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:45.590440 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" event={"ID":"f580aa78-682e-4267-a66d-c6bae2cfa5a9","Type":"ContainerStarted","Data":"ba3dfec906bd243cf7f2469b900401656e7b588c8af5bb0987c55924c0e2fde0"} Oct 30 06:35:45.590923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:45.590500 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" event={"ID":"f580aa78-682e-4267-a66d-c6bae2cfa5a9","Type":"ContainerStarted","Data":"82c4e013897a09cb34f482c895c67f8eadf256cb5b79308aa75134f887e7299b"} Oct 30 06:35:45.634648 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:45.634600 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" podStartSLOduration=1.634587453 podStartE2EDuration="1.634587453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:45.633724357 +0000 UTC m=+3804.074101551" watchObservedRunningTime="2026-10-30 06:35:45.634587453 +0000 UTC m=+3804.074964646" Oct 30 06:35:46.596963 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:46.596927 6171 generic.go:356] "Generic (PLEG): container finished" podID="f580aa78-682e-4267-a66d-c6bae2cfa5a9" containerID="ba3dfec906bd243cf7f2469b900401656e7b588c8af5bb0987c55924c0e2fde0" exitCode=0 Oct 30 06:35:46.597365 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:46.597015 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" event={"ID":"f580aa78-682e-4267-a66d-c6bae2cfa5a9","Type":"ContainerDied","Data":"ba3dfec906bd243cf7f2469b900401656e7b588c8af5bb0987c55924c0e2fde0"} Oct 30 06:35:47.030619 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.030589 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2"] Oct 30 06:35:47.033574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.033552 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-j8wv2"] Oct 30 06:35:47.671747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.671710 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e" path="/var/lib/kubelet/pods/6d64e6e9-0ba1-47fa-afc0-d5424dfffc9e/volumes" Oct 30 06:35:47.742155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.742130 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:47.765081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.765037 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\") pod \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " Oct 30 06:35:47.765162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.765119 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\") pod \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " Oct 30 06:35:47.765215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.765185 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\" (UniqueName: \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\") pod \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\" (UID: \"f580aa78-682e-4267-a66d-c6bae2cfa5a9\") " Oct 30 06:35:47.765506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.765487 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca" pod "f580aa78-682e-4267-a66d-c6bae2cfa5a9" (UID: "f580aa78-682e-4267-a66d-c6bae2cfa5a9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:47.767191 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.767167 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca" pod "f580aa78-682e-4267-a66d-c6bae2cfa5a9" (UID: "f580aa78-682e-4267-a66d-c6bae2cfa5a9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:47.767362 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.767340 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2" pod "f580aa78-682e-4267-a66d-c6bae2cfa5a9" (UID: "f580aa78-682e-4267-a66d-c6bae2cfa5a9"). InnerVolumeSpecName "kube-api-access-64md2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:47.866948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.866866 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64md2\" (UniqueName: \"kubernetes.io/projected/f580aa78-682e-4267-a66d-c6bae2cfa5a9-kube-api-access-64md2\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.866948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.866896 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:47.866948 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:47.866907 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f580aa78-682e-4267-a66d-c6bae2cfa5a9-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:35:48.609413 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:48.609378 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" Oct 30 06:35:48.609629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:48.609410 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj" event={"ID":"f580aa78-682e-4267-a66d-c6bae2cfa5a9","Type":"ContainerDied","Data":"82c4e013897a09cb34f482c895c67f8eadf256cb5b79308aa75134f887e7299b"} Oct 30 06:35:48.609629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:35:48.609462 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82c4e013897a09cb34f482c895c67f8eadf256cb5b79308aa75134f887e7299b" Oct 30 06:35:56.390866 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:35:56.390829 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:36:00.922945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.922907 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:36:00.923596 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.923574 6171 cpu_manager.go:414] "containerMap: removing container" podUID="f580aa78-682e-4267-a66d-c6bae2cfa5a9" containerName="revoke-keys" Oct 30 06:36:00.923694 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.923598 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f580aa78-682e-4267-a66d-c6bae2cfa5a9" containerName="revoke-keys" Oct 30 06:36:00.923774 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.923760 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="f580aa78-682e-4267-a66d-c6bae2cfa5a9" containerName="revoke-keys" Oct 30 06:36:00.927280 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.927252 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:00.946514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.946468 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ee1c63-serving-cert\"" Oct 30 06:36:00.946514 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.946476 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:36:00.947655 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.947631 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:36:00.960747 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.960722 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4vtg8\"" Oct 30 06:36:00.960922 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.960747 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:36:00.963103 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.963024 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:36:00.987226 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.987193 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:00.987411 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.987324 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n74w9\" (UniqueName: \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:00.987473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.987424 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:00.987473 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:00.987459 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.088427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.088389 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.088427 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.088428 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.088658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.088468 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.088658 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.088518 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n74w9\" (UniqueName: \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.089230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.089199 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.091058 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.091025 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.091143 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.091104 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.113734 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.113711 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n74w9\" (UniqueName: \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\") pod \"maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.241567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.241537 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:01.519122 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.518432 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:36:01.525205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.525173 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.542018 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.541988 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:36:01.545901 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.545866 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ee1c63\"" Oct 30 06:36:01.546097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.545926 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-72d649bc-dockercfg-7pqlq\"" Oct 30 06:36:01.592679 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.592652 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:36:01.595028 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:36:01.595001 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a99372b_6fa4_4483_a1a4_3e8a19cbf69f.slice/crio-45839451f0b9f210e83ce85d2238d7d57ad75d850101086f5a12bc79e686f28c WatchSource:0}: Error finding container 45839451f0b9f210e83ce85d2238d7d57ad75d850101086f5a12bc79e686f28c: Status 404 returned error can't find the container with id 45839451f0b9f210e83ce85d2238d7d57ad75d850101086f5a12bc79e686f28c Oct 30 06:36:01.595439 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.595416 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgnm\" (UniqueName: \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.595567 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.595543 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.596521 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.596503 6171 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:36:01.689352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.689316 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" event={"ID":"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f","Type":"ContainerStarted","Data":"45839451f0b9f210e83ce85d2238d7d57ad75d850101086f5a12bc79e686f28c"} Oct 30 06:36:01.696230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.696202 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgnm\" (UniqueName: \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.696405 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.696280 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.696937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.696918 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.721275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.721250 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgnm\" (UniqueName: \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\") pod \"payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:01.846860 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:01.846775 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:02.156654 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.156635 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:36:02.158918 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:36:02.158894 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51de6b_81f6_4015_979c_1d4fcb52db08.slice/crio-6a209c6d538d838d5463f35abda4d877473ebc275794d4f7060713272614baab WatchSource:0}: Error finding container 6a209c6d538d838d5463f35abda4d877473ebc275794d4f7060713272614baab: Status 404 returned error can't find the container with id 6a209c6d538d838d5463f35abda4d877473ebc275794d4f7060713272614baab Oct 30 06:36:02.697859 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.697813 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" event={"ID":"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f","Type":"ContainerStarted","Data":"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f"} Oct 30 06:36:02.698115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.697902 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:02.699636 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.699604 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" event={"ID":"bf51de6b-81f6-4015-979c-1d4fcb52db08","Type":"ContainerStarted","Data":"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f"} Oct 30 06:36:02.699766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.699645 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" event={"ID":"bf51de6b-81f6-4015-979c-1d4fcb52db08","Type":"ContainerStarted","Data":"6a209c6d538d838d5463f35abda4d877473ebc275794d4f7060713272614baab"} Oct 30 06:36:02.699766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.699741 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:02.797863 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.797792 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" podStartSLOduration=2.259543544 podStartE2EDuration="2.797777993s" totalImagesPullingTime="538.234449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:02.796239172 +0000 UTC m=+3821.236616365" watchObservedRunningTime="2026-10-30 06:36:02.797777993 +0000 UTC m=+3821.238155185" Oct 30 06:36:02.869275 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:02.869216 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" podStartSLOduration=1.8692021049999998 podStartE2EDuration="1.869202105s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:02.867390465 +0000 UTC m=+3821.307767655" watchObservedRunningTime="2026-10-30 06:36:02.869202105 +0000 UTC m=+3821.309579298" Oct 30 06:36:04.349408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:04.349370 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:36:08.028579 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:08.028545 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf"] Oct 30 06:36:08.042925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:08.042896 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-pz4qf"] Oct 30 06:36:08.714554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:08.714528 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:36:09.670382 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:09.670349 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24d44e9e-a3f1-4c2a-bd4a-182be7380819" path="/var/lib/kubelet/pods/24d44e9e-a3f1-4c2a-bd4a-182be7380819/volumes" Oct 30 06:36:13.711519 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:13.711493 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:36:19.143938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:19.143906 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:36:26.379364 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:36:26.379324 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:36:31.633562 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:31.633532 6171 scope.go:122] "RemoveContainer" containerID="b7788dfe3e11db22a584d1dbc135c88b697526ae2682098c54cf17a8c387c1a4" Oct 30 06:36:31.643241 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:31.643219 6171 scope.go:122] "RemoveContainer" containerID="68e3682693a4e4dd6aa15dd48d105a5d8cc5eb6b5a766f740fcc6bbc60ac06d3" Oct 30 06:36:31.651942 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:31.651924 6171 scope.go:122] "RemoveContainer" containerID="26198c05bf09eb89e94d89c991a5e7ad6950d19a6dee04035e346e837a2cd766" Oct 30 06:36:34.141381 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:34.141347 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:36:47.555989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:47.555949 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:36:56.379642 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:36:56.379594 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:36:57.580807 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:36:57.580765 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:37:00.108969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.108936 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:37:00.109535 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.109173 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" podUID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" containerName="maas-api" containerID="cri-o://f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f" gracePeriod=30 Oct 30 06:37:00.621140 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.621100 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:37:00.707099 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.707021 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\" (UniqueName: \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\") pod \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " Oct 30 06:37:00.707247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.707150 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\") pod \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " Oct 30 06:37:00.707247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.707201 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\") pod \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " Oct 30 06:37:00.707247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.707232 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\") pod \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\" (UID: \"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f\") " Oct 30 06:37:00.707723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.707687 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca" pod "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" (UID: "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:00.709412 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.709387 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls" pod "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" (UID: "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:37:00.709969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.709944 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs" pod "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" (UID: "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:37:00.710097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.709986 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9" pod "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" (UID: "0a99372b-6fa4-4483-a1a4-3e8a19cbf69f"). InnerVolumeSpecName "kube-api-access-n74w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:00.808328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.808297 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:00.808328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.808323 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:00.808328 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.808333 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:00.808559 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:00.808342 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n74w9\" (UniqueName: \"kubernetes.io/projected/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f-kube-api-access-n74w9\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:01.028937 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.028899 6171 generic.go:356] "Generic (PLEG): container finished" podID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" containerID="f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f" exitCode=0 Oct 30 06:37:01.029164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.028950 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" Oct 30 06:37:01.029164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.028988 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" event={"ID":"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f","Type":"ContainerDied","Data":"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f"} Oct 30 06:37:01.029164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.029082 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f" event={"ID":"0a99372b-6fa4-4483-a1a4-3e8a19cbf69f","Type":"ContainerDied","Data":"45839451f0b9f210e83ce85d2238d7d57ad75d850101086f5a12bc79e686f28c"} Oct 30 06:37:01.029164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.029117 6171 scope.go:122] "RemoveContainer" containerID="f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f" Oct 30 06:37:01.039425 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.039401 6171 scope.go:122] "RemoveContainer" containerID="f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f" Oct 30 06:37:01.039679 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:37:01.039660 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f\": container with ID starting with f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f not found: ID does not exist" containerID="f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f" Oct 30 06:37:01.039739 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.039689 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f"} err="failed to get container status \"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f\": rpc error: code = NotFound desc = could not find container \"f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f\": container with ID starting with f948e0f9aface069ec490306f93cc8c5d864630b94f80f23c4fe5c5919cbc57f not found: ID does not exist" Oct 30 06:37:01.086822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.086792 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:37:01.095998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.095976 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ee1c63-fd6df5d96-bv76f"] Oct 30 06:37:01.671383 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:01.671350 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" path="/var/lib/kubelet/pods/0a99372b-6fa4-4483-a1a4-3e8a19cbf69f/volumes" Oct 30 06:37:02.422074 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:02.421659 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:37:02.581934 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:02.581889 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" containerID="cri-o://1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f" gracePeriod=25 Oct 30 06:37:03.710360 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:03.710319 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:37:13.709475 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:13.709428 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:37:14.351638 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:14.351601 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:37:22.838760 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.838698 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:37:22.854876 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.854856 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\" (UniqueName: \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\") pod \"bf51de6b-81f6-4015-979c-1d4fcb52db08\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " Oct 30 06:37:22.854953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.854894 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\") pod \"bf51de6b-81f6-4015-979c-1d4fcb52db08\" (UID: \"bf51de6b-81f6-4015-979c-1d4fcb52db08\") " Oct 30 06:37:22.855343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.855319 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume" pod "bf51de6b-81f6-4015-979c-1d4fcb52db08" (UID: "bf51de6b-81f6-4015-979c-1d4fcb52db08"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:22.857146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.857121 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm" pod "bf51de6b-81f6-4015-979c-1d4fcb52db08" (UID: "bf51de6b-81f6-4015-979c-1d4fcb52db08"). InnerVolumeSpecName "kube-api-access-2dgnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:22.956107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.956078 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dgnm\" (UniqueName: \"kubernetes.io/projected/bf51de6b-81f6-4015-979c-1d4fcb52db08-kube-api-access-2dgnm\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:22.956107 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:22.956103 6171 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf51de6b-81f6-4015-979c-1d4fcb52db08-plugins-config-volume\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:23.191544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.191440 6171 generic.go:356] "Generic (PLEG): container finished" podID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerID="1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f" exitCode=0 Oct 30 06:37:23.191544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.191504 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" Oct 30 06:37:23.191544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.191525 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" event={"ID":"bf51de6b-81f6-4015-979c-1d4fcb52db08","Type":"ContainerDied","Data":"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f"} Oct 30 06:37:23.191889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.191598 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c" event={"ID":"bf51de6b-81f6-4015-979c-1d4fcb52db08","Type":"ContainerDied","Data":"6a209c6d538d838d5463f35abda4d877473ebc275794d4f7060713272614baab"} Oct 30 06:37:23.191889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.191620 6171 scope.go:122] "RemoveContainer" containerID="1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f" Oct 30 06:37:23.202663 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.202643 6171 scope.go:122] "RemoveContainer" containerID="1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f" Oct 30 06:37:23.202926 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:37:23.202906 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f\": container with ID starting with 1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f not found: ID does not exist" containerID="1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f" Oct 30 06:37:23.202996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.202932 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f"} err="failed to get container status \"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f\": rpc error: code = NotFound desc = could not find container \"1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f\": container with ID starting with 1a1e241c977d9fd58acb5856f46bfa011ddc66f722de5287c9aaa48aefac679f not found: ID does not exist" Oct 30 06:37:23.227493 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:37:23.227473 6171 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf51de6b_81f6_4015_979c_1d4fcb52db08.slice\": RecentStats: unable to find data in memory cache]" Oct 30 06:37:23.248286 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.248261 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:37:23.263147 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.263126 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ee1c63-7f9886468wlv2c"] Oct 30 06:37:23.670182 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:23.670018 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" path="/var/lib/kubelet/pods/bf51de6b-81f6-4015-979c-1d4fcb52db08/volumes" Oct 30 06:37:26.379296 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:37:26.379256 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:37:31.233057 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:31.232999 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:37:31.682980 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:31.682899 6171 scope.go:122] "RemoveContainer" containerID="f0f35b2accd81576570a9881a4f7873125d887aa4884bc89e7026f8fe6762a4c" Oct 30 06:37:31.692965 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:31.692947 6171 scope.go:122] "RemoveContainer" containerID="bf16b7523f5ec755145b012e38683a31a20c3766ccba2b6fd91f250d36c1bec0" Oct 30 06:37:31.702508 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:31.702488 6171 scope.go:122] "RemoveContainer" containerID="7e3760eef9268de1e8ec1747ec1a02aa27467898bbe76c8d3500903769975d2d" Oct 30 06:37:34.377070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:34.376250 6171 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-58847f9fc5-nrm2h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:37:34.377070 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:34.376367 6171 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-58847f9fc5-nrm2h" podUID="6eba797a-df5c-42d7-805d-26407499edc7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.190:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:37:45.539674 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.539633 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h"] Oct 30 06:37:45.540116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540055 6171 cpu_manager.go:414] "containerMap: removing container" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" Oct 30 06:37:45.540116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540067 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" Oct 30 06:37:45.540116 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540114 6171 cpu_manager.go:414] "containerMap: removing container" podUID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" containerName="maas-api" Oct 30 06:37:45.540417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540120 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" containerName="maas-api" Oct 30 06:37:45.540417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540200 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a99372b-6fa4-4483-a1a4-3e8a19cbf69f" containerName="maas-api" Oct 30 06:37:45.540417 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.540213 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf51de6b-81f6-4015-979c-1d4fcb52db08" containerName="payload-processing" Oct 30 06:37:45.544751 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.544735 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.561497 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.561471 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:37:45.575652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.575620 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:37:45.575652 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.575645 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:37:45.575889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.575635 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:37:45.578081 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.578036 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h"] Oct 30 06:37:45.628660 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.628639 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljbh4\" (UniqueName: \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.628803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.628671 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.628803 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.628760 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.729314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.729283 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.729504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.729409 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljbh4\" (UniqueName: \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.729504 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.729451 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.730118 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.730095 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.731789 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.731772 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.755162 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.755143 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljbh4\" (UniqueName: \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\") pod \"maas-api-revoke-sub-452b6d7b-nzv9h\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:45.858463 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:45.858391 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:46.159749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:46.159718 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h"] Oct 30 06:37:46.161690 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:37:46.161665 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda93a3cd4_3097_4fba_ad0b_9646ddd2103f.slice/crio-b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc WatchSource:0}: Error finding container b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc: Status 404 returned error can't find the container with id b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc Oct 30 06:37:46.373472 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:46.373430 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" event={"ID":"a93a3cd4-3097-4fba-ad0b-9646ddd2103f","Type":"ContainerStarted","Data":"b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc"} Oct 30 06:37:46.734128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:46.734092 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:37:47.384400 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:47.384293 6171 generic.go:356] "Generic (PLEG): container finished" podID="a93a3cd4-3097-4fba-ad0b-9646ddd2103f" containerID="4d465804086cfe110e3a346d597ecc9c417ea4733d763be6fa4c11eb8425ed35" exitCode=0 Oct 30 06:37:47.384599 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:47.384456 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" event={"ID":"a93a3cd4-3097-4fba-ad0b-9646ddd2103f","Type":"ContainerDied","Data":"4d465804086cfe110e3a346d597ecc9c417ea4733d763be6fa4c11eb8425ed35"} Oct 30 06:37:49.101356 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.101327 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:49.131653 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.131618 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\") pod \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " Oct 30 06:37:49.131910 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.131888 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\") pod \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " Oct 30 06:37:49.132165 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.132144 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\" (UniqueName: \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\") pod \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\" (UID: \"a93a3cd4-3097-4fba-ad0b-9646ddd2103f\") " Oct 30 06:37:49.137513 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.137489 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca" pod "a93a3cd4-3097-4fba-ad0b-9646ddd2103f" (UID: "a93a3cd4-3097-4fba-ad0b-9646ddd2103f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:37:49.164234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.164193 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4" pod "a93a3cd4-3097-4fba-ad0b-9646ddd2103f" (UID: "a93a3cd4-3097-4fba-ad0b-9646ddd2103f"). InnerVolumeSpecName "kube-api-access-ljbh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:37:49.179486 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.179313 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca" pod "a93a3cd4-3097-4fba-ad0b-9646ddd2103f" (UID: "a93a3cd4-3097-4fba-ad0b-9646ddd2103f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:37:49.234705 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.234674 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljbh4\" (UniqueName: \"kubernetes.io/projected/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-kube-api-access-ljbh4\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:49.240119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.240097 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:49.248149 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.248092 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a93a3cd4-3097-4fba-ad0b-9646ddd2103f-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:37:49.499997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.499903 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" event={"ID":"a93a3cd4-3097-4fba-ad0b-9646ddd2103f","Type":"ContainerDied","Data":"b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc"} Oct 30 06:37:49.500335 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.500312 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80b5b6d81b9bf8c5658917f632cb015f29cc315bf7d11bca4b17eb2e4b454bc" Oct 30 06:37:49.500575 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:49.500556 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-nzv9h" Oct 30 06:37:56.381135 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:37:56.381096 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:37:58.622697 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:37:58.622655 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:03.266810 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.266765 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz"] Oct 30 06:38:03.267656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.267630 6171 cpu_manager.go:414] "containerMap: removing container" podUID="a93a3cd4-3097-4fba-ad0b-9646ddd2103f" containerName="revoke-keys" Oct 30 06:38:03.267656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.267656 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a93a3cd4-3097-4fba-ad0b-9646ddd2103f" containerName="revoke-keys" Oct 30 06:38:03.267878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.267808 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="a93a3cd4-3097-4fba-ad0b-9646ddd2103f" containerName="revoke-keys" Oct 30 06:38:03.271688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.271663 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.294263 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.294234 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz"] Oct 30 06:38:03.301893 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.301865 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:38:03.302309 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.302276 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:38:03.302554 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.302537 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:38:03.305452 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.305337 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:38:03.339034 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.338996 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.339331 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.339306 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.339516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.339496 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlclt\" (UniqueName: \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.440721 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.440669 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.441063 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.441010 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.441216 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.441146 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlclt\" (UniqueName: \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.441626 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.441591 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.444314 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.444288 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.466900 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.466863 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlclt\" (UniqueName: \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\") pod \"maas-api-revoke-sub-3e31a506-b2wgz\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.597962 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.597853 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:03.970369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:03.970249 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz"] Oct 30 06:38:03.987486 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:38:03.986472 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b66ceab_ac4d_46c7_b5fd_a09c5440c0dc.slice/crio-7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229 WatchSource:0}: Error finding container 7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229: Status 404 returned error can't find the container with id 7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229 Oct 30 06:38:04.630132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:04.629510 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" event={"ID":"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc","Type":"ContainerStarted","Data":"5c6a6d96f3b2963b5f780e4c78ce5696b9e611512697942c9a96dd006920804c"} Oct 30 06:38:04.630132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:04.629583 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" event={"ID":"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc","Type":"ContainerStarted","Data":"7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229"} Oct 30 06:38:04.685077 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:04.681797 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" podStartSLOduration=1.6817786319999999 podStartE2EDuration="1.681778632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:04.681464772 +0000 UTC m=+3943.121841965" watchObservedRunningTime="2026-10-30 06:38:04.681778632 +0000 UTC m=+3943.122155829" Oct 30 06:38:05.358071 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:05.320844 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:05.671952 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:05.671909 6171 generic.go:356] "Generic (PLEG): container finished" podID="6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" containerID="5c6a6d96f3b2963b5f780e4c78ce5696b9e611512697942c9a96dd006920804c" exitCode=0 Oct 30 06:38:05.676698 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:05.676648 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" event={"ID":"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc","Type":"ContainerDied","Data":"5c6a6d96f3b2963b5f780e4c78ce5696b9e611512697942c9a96dd006920804c"} Oct 30 06:38:06.815386 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.815363 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:06.932707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.932630 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\") pod \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " Oct 30 06:38:06.932707 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.932703 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\") pod \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " Oct 30 06:38:06.932938 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.932751 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\" (UniqueName: \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\") pod \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\" (UID: \"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc\") " Oct 30 06:38:06.932987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.932960 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca" pod "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" (UID: "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:06.933203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.933182 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:06.935059 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.935017 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca" pod "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" (UID: "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:06.935175 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:06.935039 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt" pod "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" (UID: "6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc"). InnerVolumeSpecName "kube-api-access-zlclt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:07.033914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:07.033880 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:07.033914 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:07.033910 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlclt\" (UniqueName: \"kubernetes.io/projected/6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc-kube-api-access-zlclt\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:07.686173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:07.686141 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" event={"ID":"6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc","Type":"ContainerDied","Data":"7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229"} Oct 30 06:38:07.686173 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:07.686163 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-b2wgz" Oct 30 06:38:07.686374 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:07.686189 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bfc82a4d024d3d7a7dcda0d0c402b91e7c749c89ce00267fe13f4615c795229" Oct 30 06:38:14.530821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:14.530788 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:21.262343 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:21.262306 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:26.381238 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:38:26.381198 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:38:26.678474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.678398 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm"] Oct 30 06:38:26.678819 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.678807 6171 cpu_manager.go:414] "containerMap: removing container" podUID="6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" containerName="revoke-keys" Oct 30 06:38:26.678869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.678821 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" containerName="revoke-keys" Oct 30 06:38:26.678908 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.678902 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc" containerName="revoke-keys" Oct 30 06:38:26.682136 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.682119 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.702387 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.702360 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm"] Oct 30 06:38:26.713151 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.713129 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:38:26.713266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.713150 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:38:26.713266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.713129 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:38:26.713266 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.713220 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7z5tn\"" Oct 30 06:38:26.788685 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.788653 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.788850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.788706 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf645\" (UniqueName: \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.788850 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.788731 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.890321 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.890290 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf645\" (UniqueName: \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.890526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.890331 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.890526 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.890465 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.891039 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.891015 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.892983 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.892960 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.915488 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.915466 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf645\" (UniqueName: \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\") pod \"maas-api-revoke-sub-0c3684cc-fxtdm\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:26.995268 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:26.995247 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:27.293248 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:27.293214 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm"] Oct 30 06:38:27.294534 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:38:27.294504 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod704cd6a1_3b77_4ed1_904a_2c6ce3228843.slice/crio-26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1 WatchSource:0}: Error finding container 26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1: Status 404 returned error can't find the container with id 26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1 Oct 30 06:38:27.885923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:27.885822 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" event={"ID":"704cd6a1-3b77-4ed1-904a-2c6ce3228843","Type":"ContainerStarted","Data":"cf638f39999b41b6411a92b0fae0291767f2dcf131cd51c866f97b5fc2725514"} Oct 30 06:38:27.885923 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:27.885895 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" event={"ID":"704cd6a1-3b77-4ed1-904a-2c6ce3228843","Type":"ContainerStarted","Data":"26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1"} Oct 30 06:38:27.930073 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:27.930006 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" podStartSLOduration=1.929992968 podStartE2EDuration="1.929992968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:27.92878732 +0000 UTC m=+3966.369164512" watchObservedRunningTime="2026-10-30 06:38:27.929992968 +0000 UTC m=+3966.370370161" Oct 30 06:38:28.892111 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:28.892077 6171 generic.go:356] "Generic (PLEG): container finished" podID="704cd6a1-3b77-4ed1-904a-2c6ce3228843" containerID="cf638f39999b41b6411a92b0fae0291767f2dcf131cd51c866f97b5fc2725514" exitCode=0 Oct 30 06:38:28.892532 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:28.892156 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" event={"ID":"704cd6a1-3b77-4ed1-904a-2c6ce3228843","Type":"ContainerDied","Data":"cf638f39999b41b6411a92b0fae0291767f2dcf131cd51c866f97b5fc2725514"} Oct 30 06:38:30.036947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.036924 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:30.119369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.119332 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\" (UniqueName: \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\") pod \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " Oct 30 06:38:30.119369 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.119373 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\") pod \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " Oct 30 06:38:30.119584 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.119468 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\") pod \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\" (UID: \"704cd6a1-3b77-4ed1-904a-2c6ce3228843\") " Oct 30 06:38:30.119841 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.119804 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca" pod "704cd6a1-3b77-4ed1-904a-2c6ce3228843" (UID: "704cd6a1-3b77-4ed1-904a-2c6ce3228843"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:30.121630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.121603 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca" pod "704cd6a1-3b77-4ed1-904a-2c6ce3228843" (UID: "704cd6a1-3b77-4ed1-904a-2c6ce3228843"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:30.121748 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.121663 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645" pod "704cd6a1-3b77-4ed1-904a-2c6ce3228843" (UID: "704cd6a1-3b77-4ed1-904a-2c6ce3228843"). InnerVolumeSpecName "kube-api-access-jf645". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:30.220766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.220688 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-xks-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:30.220766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.220714 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf645\" (UniqueName: \"kubernetes.io/projected/704cd6a1-3b77-4ed1-904a-2c6ce3228843-kube-api-access-jf645\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:30.220766 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.220725 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/704cd6a1-3b77-4ed1-904a-2c6ce3228843-maas-api-openshift-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:30.905037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.904995 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" event={"ID":"704cd6a1-3b77-4ed1-904a-2c6ce3228843","Type":"ContainerDied","Data":"26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1"} Oct 30 06:38:30.905219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.905009 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c3684cc-fxtdm" Oct 30 06:38:30.905219 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:30.905068 6171 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f24ae55faf97812462fc60ad3f79365923ac13a65eb6ce55088fad2caf79f1" Oct 30 06:38:31.738951 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:31.738866 6171 scope.go:122] "RemoveContainer" containerID="84410bd273d9c013b7f1e2f2762582ee3b74bc0b83af1885934451bf000adc29" Oct 30 06:38:31.750228 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:31.750203 6171 scope.go:122] "RemoveContainer" containerID="6f7e3ad902b4c7335add2229f938861930f932bd9440d986ec83c959bd462731" Oct 30 06:38:37.842818 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:37.842787 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:38.868131 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.868102 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:38.868737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.868718 6171 cpu_manager.go:414] "containerMap: removing container" podUID="704cd6a1-3b77-4ed1-904a-2c6ce3228843" containerName="revoke-keys" Oct 30 06:38:38.868821 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.868740 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="704cd6a1-3b77-4ed1-904a-2c6ce3228843" containerName="revoke-keys" Oct 30 06:38:38.868879 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.868835 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="704cd6a1-3b77-4ed1-904a-2c6ce3228843" containerName="revoke-keys" Oct 30 06:38:38.873515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.873499 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:38.896155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.896119 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:38.905916 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.905877 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:38:38.906098 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.905973 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:38:38.906688 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.906618 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:38:38.906907 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.906888 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-4vtg8\"" Oct 30 06:38:38.906977 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:38.906901 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 06:38:39.002830 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.002797 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4c6g\" (UniqueName: \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.002997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.002850 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.002997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.002974 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.003115 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.003010 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.104332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.104303 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.104516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.104337 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.104570 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.104533 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4c6g\" (UniqueName: \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.104609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.104599 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.105029 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.105006 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.106905 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.106884 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.107142 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.107123 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.129656 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.129604 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4c6g\" (UniqueName: \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\") pod \"maas-api-5c48877c75-2pbs6\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.189629 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.189602 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:39.504737 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.504709 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:39.505702 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:38:39.505661 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57c8fb4f_40e0_4039_9b98_89f0b8b1f960.slice/crio-93577be77ce49ec3fda5b88a88495d74da60f9cc98139d4604ba6cded06d27a8 WatchSource:0}: Error finding container 93577be77ce49ec3fda5b88a88495d74da60f9cc98139d4604ba6cded06d27a8: Status 404 returned error can't find the container with id 93577be77ce49ec3fda5b88a88495d74da60f9cc98139d4604ba6cded06d27a8 Oct 30 06:38:39.964146 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:39.964112 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" event={"ID":"57c8fb4f-40e0-4039-9b98-89f0b8b1f960","Type":"ContainerStarted","Data":"93577be77ce49ec3fda5b88a88495d74da60f9cc98139d4604ba6cded06d27a8"} Oct 30 06:38:40.970964 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:40.970925 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" event={"ID":"57c8fb4f-40e0-4039-9b98-89f0b8b1f960","Type":"ContainerStarted","Data":"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33"} Oct 30 06:38:40.971418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:40.971012 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:41.023375 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:41.023331 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" podStartSLOduration=2.5728074 podStartE2EDuration="3.023319204s" totalImagesPullingTime="450.511804ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:41.022390114 +0000 UTC m=+3979.462767307" watchObservedRunningTime="2026-10-30 06:38:41.023319204 +0000 UTC m=+3979.463696396" Oct 30 06:38:43.037791 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:43.032831 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz"] Oct 30 06:38:43.044205 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:43.044178 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qd9fz"] Oct 30 06:38:43.670959 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:43.670925 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a077f5f-994c-4474-af7f-45baf9bf3430" path="/var/lib/kubelet/pods/3a077f5f-994c-4474-af7f-45baf9bf3430/volumes" Oct 30 06:38:46.106485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.106453 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:46.106957 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.106639 6171 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" podUID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" containerName="maas-api" containerID="cri-o://b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33" gracePeriod=30 Oct 30 06:38:46.111889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.111866 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:46.144332 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.144306 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gmtg7"] Oct 30 06:38:46.147878 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.147863 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.169849 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.169818 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-maas-api-tls\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.169972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.169904 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-metrics-certs\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.169972 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.169939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj27k\" (UniqueName: \"kubernetes.io/projected/5a08479e-78e2-4777-8695-263d18c1b823-kube-api-access-lj27k\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.170223 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.170188 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a08479e-78e2-4777-8695-263d18c1b823-openshift-service-ca\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.172256 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.172231 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gmtg7"] Oct 30 06:38:46.271255 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.271225 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a08479e-78e2-4777-8695-263d18c1b823-openshift-service-ca\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.271418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.271277 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-maas-api-tls\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.271418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.271326 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-metrics-certs\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.271418 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.271346 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj27k\" (UniqueName: \"kubernetes.io/projected/5a08479e-78e2-4777-8695-263d18c1b823-kube-api-access-lj27k\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.272096 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.272066 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a08479e-78e2-4777-8695-263d18c1b823-openshift-service-ca\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.273996 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.273971 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-maas-api-tls\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.274197 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.274178 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5a08479e-78e2-4777-8695-263d18c1b823-metrics-certs\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.297021 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.296991 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj27k\" (UniqueName: \"kubernetes.io/projected/5a08479e-78e2-4777-8695-263d18c1b823-kube-api-access-lj27k\") pod \"maas-api-595cc648c-gmtg7\" (UID: \"5a08479e-78e2-4777-8695-263d18c1b823\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.379869 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.379845 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:46.472109 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.472077 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\") pod \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " Oct 30 06:38:46.472260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.472168 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\" (UniqueName: \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\") pod \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " Oct 30 06:38:46.472260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.472223 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\") pod \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " Oct 30 06:38:46.472260 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.472255 6171 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\") pod \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\" (UID: \"57c8fb4f-40e0-4039-9b98-89f0b8b1f960\") " Oct 30 06:38:46.472642 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.472616 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca" pod "57c8fb4f-40e0-4039-9b98-89f0b8b1f960" (UID: "57c8fb4f-40e0-4039-9b98-89f0b8b1f960"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:38:46.474347 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.474316 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls" pod "57c8fb4f-40e0-4039-9b98-89f0b8b1f960" (UID: "57c8fb4f-40e0-4039-9b98-89f0b8b1f960"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:46.474515 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.474497 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs" pod "57c8fb4f-40e0-4039-9b98-89f0b8b1f960" (UID: "57c8fb4f-40e0-4039-9b98-89f0b8b1f960"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:38:46.474574 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.474555 6171 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g" pod "57c8fb4f-40e0-4039-9b98-89f0b8b1f960" (UID: "57c8fb4f-40e0-4039-9b98-89f0b8b1f960"). InnerVolumeSpecName "kube-api-access-l4c6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:38:46.504530 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.504507 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:46.574101 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.574068 6171 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-maas-api-tls\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:46.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.574108 6171 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4c6g\" (UniqueName: \"kubernetes.io/projected/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-kube-api-access-l4c6g\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:46.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.574123 6171 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-openshift-service-ca\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:46.574234 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.574136 6171 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57c8fb4f-40e0-4039-9b98-89f0b8b1f960-metrics-certs\") on node \"ip-10-0-136-35.ec2.internal\" DevicePath \"\"" Oct 30 06:38:46.804630 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:46.804603 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-gmtg7"] Oct 30 06:38:46.805985 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:38:46.805958 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a08479e_78e2_4777_8695_263d18c1b823.slice/crio-5a927c2301cd37960346264141d9e4244bf91297c1ba5b423161b55252bedbc7 WatchSource:0}: Error finding container 5a927c2301cd37960346264141d9e4244bf91297c1ba5b423161b55252bedbc7: Status 404 returned error can't find the container with id 5a927c2301cd37960346264141d9e4244bf91297c1ba5b423161b55252bedbc7 Oct 30 06:38:47.004506 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.004469 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" event={"ID":"5a08479e-78e2-4777-8695-263d18c1b823","Type":"ContainerStarted","Data":"5a927c2301cd37960346264141d9e4244bf91297c1ba5b423161b55252bedbc7"} Oct 30 06:38:47.005811 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.005786 6171 generic.go:356] "Generic (PLEG): container finished" podID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" containerID="b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33" exitCode=0 Oct 30 06:38:47.005919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.005834 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" event={"ID":"57c8fb4f-40e0-4039-9b98-89f0b8b1f960","Type":"ContainerDied","Data":"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33"} Oct 30 06:38:47.005919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.005865 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" event={"ID":"57c8fb4f-40e0-4039-9b98-89f0b8b1f960","Type":"ContainerDied","Data":"93577be77ce49ec3fda5b88a88495d74da60f9cc98139d4604ba6cded06d27a8"} Oct 30 06:38:47.005919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.005885 6171 scope.go:122] "RemoveContainer" containerID="b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33" Oct 30 06:38:47.005919 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.005835 6171 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6" Oct 30 06:38:47.016577 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.016555 6171 scope.go:122] "RemoveContainer" containerID="b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33" Oct 30 06:38:47.016859 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:38:47.016841 6171 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33\": container with ID starting with b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33 not found: ID does not exist" containerID="b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33" Oct 30 06:38:47.016903 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.016870 6171 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33"} err="failed to get container status \"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33\": rpc error: code = NotFound desc = could not find container \"b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33\": container with ID starting with b6395e99d9f46d60b3ea9e39f418b6ed85563fe4fe081b6c26fdb8246724ef33 not found: ID does not exist" Oct 30 06:38:47.064037 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.063979 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:47.067476 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.067457 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-2pbs6"] Oct 30 06:38:47.425512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.425434 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:47.670031 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:47.670000 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" path="/var/lib/kubelet/pods/57c8fb4f-40e0-4039-9b98-89f0b8b1f960/volumes" Oct 30 06:38:48.045998 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:48.045946 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" event={"ID":"5a08479e-78e2-4777-8695-263d18c1b823","Type":"ContainerStarted","Data":"cd8db9a68bd586ac09b64c7c7de9edaa391f5d2fb5fdd3da174ab7c94a07f712"} Oct 30 06:38:48.046870 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:48.046845 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:48.098023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:48.097965 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" podStartSLOduration=1.799530337 podStartE2EDuration="2.097946704s" totalImagesPullingTime="298.416367ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:38:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:38:48.093064286 +0000 UTC m=+3986.533441480" watchObservedRunningTime="2026-10-30 06:38:48.097946704 +0000 UTC m=+3986.538323898" Oct 30 06:38:55.071379 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:55.071347 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-gmtg7" Oct 30 06:38:55.540015 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:38:55.539977 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:38:56.393018 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:38:56.392961 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:39:09.132245 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:09.132207 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:39:26.379875 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:39:26.379838 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:39:37.056809 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:37.056774 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-h82bs"] Oct 30 06:39:47.675706 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:47.675672 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654df4bd7-95hcs_6e57dbc4-5ff8-4158-981f-a3b456468cb3/manager/0.log" Oct 30 06:39:48.106989 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:48.106955 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-vghzm_55458753-8f72-4e41-b495-6ce7fda91c10/manager/0.log" Oct 30 06:39:48.583120 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:48.583089 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:39:49.209230 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:49.209194 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-7nfbb_b1f10f8e-96a4-4c21-9175-31da2899dee8/manager/0.log" Oct 30 06:39:50.208155 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:50.208124 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-z5lzv_465f403f-78c3-4195-9fbd-cd25c839fb2e/copy-manifests/0.log" Oct 30 06:39:50.239666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:50.239637 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-z5lzv_465f403f-78c3-4195-9fbd-cd25c839fb2e/manager/0.log" Oct 30 06:39:51.411564 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:51.411534 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-gmtg7_5a08479e-78e2-4777-8695-263d18c1b823/maas-api/0.log" Oct 30 06:39:51.807673 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:51.807644 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0c3684cc-fxtdm_704cd6a1-3b77-4ed1-904a-2c6ce3228843/revoke-keys/0.log" Oct 30 06:39:51.998204 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:51.998177 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-b2wgz_6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc/revoke-keys/0.log" Oct 30 06:39:52.186027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:52.185942 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nzv9h_a93a3cd4-3097-4fba-ad0b-9646ddd2103f/revoke-keys/0.log" Oct 30 06:39:52.732185 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:52.732157 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4lcmj_f580aa78-682e-4267-a66d-c6bae2cfa5a9/revoke-keys/0.log" Oct 30 06:39:52.901516 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:52.901484 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-p985h_113d462d-8e7f-4bc9-8cb2-5d4f24028098/revoke-keys/0.log" Oct 30 06:39:55.872973 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:55.872940 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h82bs_d3509408-6100-43da-8ff8-fd3b5768bcaa/limitador/0.log" Oct 30 06:39:56.083675 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:56.083638 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-f8dcg_cc918c8e-0360-430e-a745-2dd3588e0328/manager/0.log" Oct 30 06:39:56.379966 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:39:56.379926 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:39:57.142889 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:57.142859 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rdsvq_e7285465-20bc-47c8-82dc-b4baa7f4b38b/discovery/0.log" Oct 30 06:39:57.359269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:57.359240 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8596b47b87-7k748_d8d4e9ee-4fee-4c4b-8187-e7a44b20e101/kube-auth-proxy/0.log" Oct 30 06:39:58.103932 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:58.103895 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-84wh7_b3cabe1b-3080-4d86-b821-a1e469c7439e/istio-proxy/0.log" Oct 30 06:39:58.518027 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:58.517981 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mbq7p_f4a3a294-bca7-4761-8947-f0b5d4125a72/istio-proxy/0.log" Oct 30 06:39:58.926104 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:58.926011 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-nrm2h_6eba797a-df5c-42d7-805d-26407499edc7/istio-proxy/0.log" Oct 30 06:39:59.338640 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:59.338611 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-rxgds_b78e85c7-fea8-49e0-af5a-802c1ebe3ea9/istio-proxy/0.log" Oct 30 06:39:59.905242 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:39:59.905198 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:40:01.213240 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:01.213209 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-crblg_6c89f1f8-53fb-4c78-b610-f0709cd448a5/main/0.log" Oct 30 06:40:01.418215 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:01.418139 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-q74rp_3e622128-5588-4764-91dd-f7759834b7eb/main/0.log" Oct 30 06:40:01.620128 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:01.620100 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7gh9t_9c5b4302-435f-4bc0-8578-73fa12bdb419/main/0.log" Oct 30 06:40:02.026288 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:02.026259 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t_653bfbea-50ba-4751-b13e-da9ef57544bd/main/0.log" Oct 30 06:40:13.579213 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:13.579176 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-654df4bd7-95hcs_6e57dbc4-5ff8-4158-981f-a3b456468cb3/manager/0.log" Oct 30 06:40:13.992110 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:13.992081 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-5cfb77bf8c-vghzm_55458753-8f72-4e41-b495-6ce7fda91c10/manager/0.log" Oct 30 06:40:14.422361 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:14.422278 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:40:15.049119 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:15.049083 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-7nfbb_b1f10f8e-96a4-4c21-9175-31da2899dee8/manager/0.log" Oct 30 06:40:15.738680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:15.738651 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-z5lzv_465f403f-78c3-4195-9fbd-cd25c839fb2e/copy-manifests/0.log" Oct 30 06:40:15.766939 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:15.766912 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-f7d897f7-z5lzv_465f403f-78c3-4195-9fbd-cd25c839fb2e/manager/0.log" Oct 30 06:40:16.676943 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:16.676902 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-gmtg7_5a08479e-78e2-4777-8695-263d18c1b823/maas-api/0.log" Oct 30 06:40:17.057806 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:17.057780 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-0c3684cc-fxtdm_704cd6a1-3b77-4ed1-904a-2c6ce3228843/revoke-keys/0.log" Oct 30 06:40:17.241947 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:17.241920 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-b2wgz_6b66ceab-ac4d-46c7-b5fd-a09c5440c0dc/revoke-keys/0.log" Oct 30 06:40:17.423765 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:17.423682 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-nzv9h_a93a3cd4-3097-4fba-ad0b-9646ddd2103f/revoke-keys/0.log" Oct 30 06:40:17.954430 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:17.954392 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4lcmj_f580aa78-682e-4267-a66d-c6bae2cfa5a9/revoke-keys/0.log" Oct 30 06:40:18.134544 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:18.134515 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-p985h_113d462d-8e7f-4bc9-8cb2-5d4f24028098/revoke-keys/0.log" Oct 30 06:40:20.663862 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:20.663832 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h82bs_d3509408-6100-43da-8ff8-fd3b5768bcaa/limitador/0.log" Oct 30 06:40:20.853555 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:20.853525 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-f8dcg_cc918c8e-0360-430e-a745-2dd3588e0328/manager/0.log" Oct 30 06:40:21.614060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:21.614013 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-rdsvq_e7285465-20bc-47c8-82dc-b4baa7f4b38b/discovery/0.log" Oct 30 06:40:21.819247 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:21.819200 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8596b47b87-7k748_d8d4e9ee-4fee-4c4b-8187-e7a44b20e101/kube-auth-proxy/0.log" Oct 30 06:40:22.561982 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:22.561944 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-84wh7_b3cabe1b-3080-4d86-b821-a1e469c7439e/istio-proxy/0.log" Oct 30 06:40:22.975512 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:22.975479 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-mbq7p_f4a3a294-bca7-4761-8947-f0b5d4125a72/istio-proxy/0.log" Oct 30 06:40:23.364092 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:23.364003 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-nrm2h_6eba797a-df5c-42d7-805d-26407499edc7/istio-proxy/0.log" Oct 30 06:40:23.750753 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:23.750724 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-58847f9fc5-rxgds_b78e85c7-fea8-49e0-af5a-802c1ebe3ea9/istio-proxy/0.log" Oct 30 06:40:24.334080 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:24.334029 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:40:25.281282 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:25.281245 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-crblg_6c89f1f8-53fb-4c78-b610-f0709cd448a5/main/0.log" Oct 30 06:40:25.469895 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:25.469859 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-q74rp_3e622128-5588-4764-91dd-f7759834b7eb/main/0.log" Oct 30 06:40:25.661075 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:25.660969 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-7gh9t_9c5b4302-435f-4bc0-8578-73fa12bdb419/main/0.log" Oct 30 06:40:26.052680 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:26.052652 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-gct9t_653bfbea-50ba-4751-b13e-da9ef57544bd/main/0.log" Oct 30 06:40:26.380534 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:40:26.380453 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:40:31.040494 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:31.040456 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h"] Oct 30 06:40:31.054420 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:31.054391 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-p985h"] Oct 30 06:40:31.671568 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:31.671534 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="113d462d-8e7f-4bc9-8cb2-5d4f24028098" path="/var/lib/kubelet/pods/113d462d-8e7f-4bc9-8cb2-5d4f24028098/volumes" Oct 30 06:40:31.787002 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:31.786974 6171 scope.go:122] "RemoveContainer" containerID="204cdcf6b60e08e16d64af0eea1322bdcaf35fe2a37c794789aa3c8fa0c352e1" Oct 30 06:40:48.028929 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:48.028898 6171 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj"] Oct 30 06:40:48.042815 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:48.042793 6171 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4lcmj"] Oct 30 06:40:49.669749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:49.669712 6171 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f580aa78-682e-4267-a66d-c6bae2cfa5a9" path="/var/lib/kubelet/pods/f580aa78-682e-4267-a66d-c6bae2cfa5a9/volumes" Oct 30 06:40:56.380854 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:40:56.380813 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:40:56.587798 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:56.587761 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:40:56.899690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:56.899661 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:40:57.565566 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:57.565539 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:40:57.754192 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:40:57.754163 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:41:26.379803 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:41:26.379764 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:41:31.809458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:31.809383 6171 scope.go:122] "RemoveContainer" containerID="d979dbd83a105eb629bf3a901bde32013bd3cb940dd678cf79fb7137b93828aa" Oct 30 06:41:35.747855 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:35.747821 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:41:36.056985 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:36.056911 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5974d5d96-jrgzk_eb85f3e3-0044-419f-b654-9e4d1b89626b/manager/0.log" Oct 30 06:41:36.738758 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:36.738731 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:41:36.923609 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:36.923584 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-bq9hh_7e5911c2-912e-496c-b4f9-13fe71875626/payload-processing/0.log" Oct 30 06:41:41.833352 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:41.833320 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-f8w5w_5690cc6f-94f7-4bc0-b271-35c91580a7e8/global-pull-secret-syncer/0.log" Oct 30 06:41:42.927132 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:42.927100 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gnmzd_8320d815-b161-471e-97d6-dda571fef137/konnectivity-agent/0.log" Oct 30 06:41:43.308164 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:43.308126 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-35.ec2.internal_75abb82a1e3a44a283c8687c87965bd9/haproxy/0.log" Oct 30 06:41:50.763474 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:50.763440 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-h82bs_d3509408-6100-43da-8ff8-fd3b5768bcaa/limitador/0.log" Oct 30 06:41:50.956301 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:50.956268 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-856dd584c5-f8dcg_cc918c8e-0360-430e-a745-2dd3588e0328/manager/0.log" Oct 30 06:41:51.798969 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.798932 6171 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x"] Oct 30 06:41:51.799408 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.799393 6171 cpu_manager.go:414] "containerMap: removing container" podUID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" containerName="maas-api" Oct 30 06:41:51.799456 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.799408 6171 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" containerName="maas-api" Oct 30 06:41:51.799498 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.799475 6171 memory_manager.go:365] "RemoveStaleState removing state" podUID="57c8fb4f-40e0-4039-9b98-89f0b8b1f960" containerName="maas-api" Oct 30 06:41:51.803925 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.803907 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.823997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.823972 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x"] Oct 30 06:41:51.862720 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.862688 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-proc\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.862880 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.862801 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-lib-modules\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.862961 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.862939 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-sys\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.863097 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.863084 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-podres\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.863161 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.863116 6171 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-574fm\" (UniqueName: \"kubernetes.io/projected/67f026c6-765c-4959-b064-bc84ab948c86-kube-api-access-574fm\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.911768 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.911741 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-88m2b\"/\"kube-root-ca.crt\"" Oct 30 06:41:51.911873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.911787 6171 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-88m2b\"/\"openshift-service-ca.crt\"" Oct 30 06:41:51.911873 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.911822 6171 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-88m2b\"/\"default-dockercfg-hccqc\"" Oct 30 06:41:51.964158 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964132 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-proc\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964163 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-lib-modules\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964211 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-sys\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964273 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964253 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-proc\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964319 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-sys\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964334 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-lib-modules\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964321 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-podres\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964396 6171 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-574fm\" (UniqueName: \"kubernetes.io/projected/67f026c6-765c-4959-b064-bc84ab948c86-kube-api-access-574fm\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.964477 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.964364 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/67f026c6-765c-4959-b064-bc84ab948c86-podres\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:51.989752 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:51.989728 6171 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-574fm\" (UniqueName: \"kubernetes.io/projected/67f026c6-765c-4959-b064-bc84ab948c86-kube-api-access-574fm\") pod \"perf-node-gather-daemonset-6g87x\" (UID: \"67f026c6-765c-4959-b064-bc84ab948c86\") " pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:52.116945 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.116878 6171 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:52.421896 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.417886 6171 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x"] Oct 30 06:41:52.422194 ip-10-0-136-35 kubenswrapper[6171]: W1030 06:41:52.421986 6171 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod67f026c6_765c_4959_b064_bc84ab948c86.slice/crio-b62bd9188d7fb0362cef3bf70caef69a2cba59675c75503d391282c0997692cd WatchSource:0}: Error finding container b62bd9188d7fb0362cef3bf70caef69a2cba59675c75503d391282c0997692cd: Status 404 returned error can't find the container with id b62bd9188d7fb0362cef3bf70caef69a2cba59675c75503d391282c0997692cd Oct 30 06:41:52.462254 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.462223 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/alertmanager/0.log" Oct 30 06:41:52.581129 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.581102 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/config-reloader/0.log" Oct 30 06:41:52.697458 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.697397 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/kube-rbac-proxy-web/0.log" Oct 30 06:41:52.817485 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.817455 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/kube-rbac-proxy/0.log" Oct 30 06:41:52.937380 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:52.937352 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/kube-rbac-proxy-metric/0.log" Oct 30 06:41:53.054953 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.054929 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/prom-label-proxy/0.log" Oct 30 06:41:53.170006 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.169972 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_2e82198e-beb3-4f85-965d-09b144b8d640/init-config-reloader/0.log" Oct 30 06:41:53.172730 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.172704 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" event={"ID":"67f026c6-765c-4959-b064-bc84ab948c86","Type":"ContainerStarted","Data":"b20ae5281e42e964f9878e7ea8744ecc8b76ba495f5752857a7ca2ce18519bd1"} Oct 30 06:41:53.172912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.172749 6171 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" event={"ID":"67f026c6-765c-4959-b064-bc84ab948c86","Type":"ContainerStarted","Data":"b62bd9188d7fb0362cef3bf70caef69a2cba59675c75503d391282c0997692cd"} Oct 30 06:41:53.172912 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.172773 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:41:53.218548 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.218501 6171 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" podStartSLOduration=2.218487932 podStartE2EDuration="2.218487932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:41:53.217160488 +0000 UTC m=+4171.657537681" watchObservedRunningTime="2026-10-30 06:41:53.218487932 +0000 UTC m=+4171.658865124" Oct 30 06:41:53.747203 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:53.747174 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-689db8797f-sb5b8_977c05aa-1b1d-4280-bd13-a098a44bb0a7/metrics-server/0.log" Oct 30 06:41:54.322676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:54.322639 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlj4r_82812395-be27-46dd-8b16-bfab2c7fc22e/node-exporter/0.log" Oct 30 06:41:54.444145 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:54.444119 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlj4r_82812395-be27-46dd-8b16-bfab2c7fc22e/kube-rbac-proxy/0.log" Oct 30 06:41:54.562997 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:54.562976 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-nlj4r_82812395-be27-46dd-8b16-bfab2c7fc22e/init-textfile/0.log" Oct 30 06:41:56.380302 ip-10-0-136-35 kubenswrapper[6171]: E1030 06:41:56.380270 6171 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-4tpzp" containerName="extractor" Oct 30 06:41:56.533666 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:56.533638 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-665558d944-qjs57_294487e3-1f88-4803-8b40-edc9c386ad50/telemeter-client/0.log" Oct 30 06:41:56.649822 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:56.649743 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-665558d944-qjs57_294487e3-1f88-4803-8b40-edc9c386ad50/reload/0.log" Oct 30 06:41:56.765193 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:56.765163 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-665558d944-qjs57_294487e3-1f88-4803-8b40-edc9c386ad50/kube-rbac-proxy/0.log" Oct 30 06:41:56.890621 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:56.890592 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/thanos-query/0.log" Oct 30 06:41:57.005495 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:57.005469 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/kube-rbac-proxy-web/0.log" Oct 30 06:41:57.121676 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:57.121650 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/kube-rbac-proxy/0.log" Oct 30 06:41:57.239987 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:57.239962 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/prom-label-proxy/0.log" Oct 30 06:41:57.357269 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:57.357196 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/kube-rbac-proxy-rules/0.log" Oct 30 06:41:57.473363 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:57.473338 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7d56d56779-5qhql_7b75558c-ebeb-4ae6-a623-676f3daef4f5/kube-rbac-proxy-metrics/0.log" Oct 30 06:41:59.189023 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:41:59.188998 6171 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-88m2b/perf-node-gather-daemonset-6g87x" Oct 30 06:42:05.005690 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:42:05.005660 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5pf44_151d83c7-e9e4-4e60-ad4b-591771415adc/dns/0.log" Oct 30 06:42:05.122060 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:42:05.122002 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5pf44_151d83c7-e9e4-4e60-ad4b-591771415adc/kube-rbac-proxy/0.log" Oct 30 06:42:05.703723 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:42:05.703690 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-bv9np_e6c19260-4322-43b7-8529-f7c520ad4feb/dns-node-resolver/0.log" Oct 30 06:42:07.340749 ip-10-0-136-35 kubenswrapper[6171]: I1030 06:42:07.340712 6171 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hh7pw_199ee6eb-216b-471a-8195-b9a71b842c16/node-ca/0.log"